Author |
Message |
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
For everyone interested in getting the raw data returned from sensors, there is a simple script that allows synchronization with the database.
http://radioactiveathome.org/boinc/gettrickledata.php
http://radioactiveathome.org/boinc/gettrickledata.php?start=X&hostid=Y
The first line shows min:max sample ID currently available in the database, then up to 5000 lines of samples are returned.
"hostid" is optional and limits returned data to single host.
"start" skips data until last position you already have (or 0 to read from the beginning), the first returned sample ID is start+1 (or the minimum id available in the DB).
Row data format:
The live database is limited up to 31 days back (it is cleared from time to time).
There are currently 3 types of samples: "f" - first one read from sensor at the app start, "n" - normal sample taken when app is running, "r" - sample taken when the app was resumed/restarted. Both "f" and "r" may contain "long" samples taken for example when the PC was left in standby and the sensor was running.
For older data take a look here:
http://radioactiveathome.org/boinc/download/data/ |
|
|
|
Is there a formula for calculating uSv/h from the pulse count and sample time values?
____________
|
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
I believe it's (number_of_pulses/time_in_minutes)/171.232876
|
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
The script was updated, the last digit is the "experiment" flag which is set to 1 if the user marks his readings as test results (testing hardware/software/tubes etc).
____________
|
|
|
|
There appears to be a new field after the experimental flag.
____________
|
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
Convert the integer to hexadecimal to get two bytes with BCD-encoded sensor hardware/software version.
It's not fully supported yet, as it requires app update. Windows version 1.56 takes care of it, while Linux version will come later today.
|
|
|
|
Thanks, TJM.
____________
|
|
|
|
I have a problem, there are no more data is written to the last entry is 03/17/2012 23:52 why
look here http://radioactiveathome.org/boinc/gettrickledata.php?start=X&hostid=1722
lg,
Wolfgang |
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
Look at the link you provided, you have left "X" where the script expects starting position. Without knowing the position it returns first 5000 records.
|
|
|
|
Sorry, my mistake Thanks for the answer TJM
lg,
Wolfgang
____________
|
|
|
|
hi guys,
many thanks for all this info.
However, is there any simple trick (script etc.) to download the whole database or all data for single station without manual downloading or generating plenty of download links?
We would like to perform some analyses of long term data (1 month, first only for 1-2 stations) and later maybe more.
thanks much |
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
I can provide full dump of the samples database, but it's so huge, that it can't be exported regularly (takes a lot of time and bandwidth).
The best way to keep up with our data is to import the dump and then write a script that pulls fresh data from the server every now and then, it doesn't stress the server and keeps the traffic low, as only new data is downloaded on each run.
|
|
|
|
Thanks for reply TJM.
We are not going to download full dump regularly, but we would need it as start point.
I would appreciate any advice in this case.
For example if I use this link:
http://radioactiveathome.org/boinc/gettrickledata.php?start=X&hostid=Y
without any parameter set,
I get output for 48763789 to 48770832, while there is written that the database contains data 48763789:60449876
If I modify the link to:
http://radioactiveathome.org/boinc/gettrickledata.php?start=60449876&hostid=Y
I get the last portion of the data.
However, if I am right, the Sample ID (starts now at 48763789) changes, so I cannot use it as fixed value in a script.
I know, that I can jump to next data page by using the last sample id as the X parameter in the link.
Second, it would be necessary - if we consider daily database update - to somehow set the script to only load pages for the particular day.
Is there any additional parameter which could be used for this task or do I have to somehow calculate/guess the approximate number of data pages, which should be checked?
thanks |
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
To pull fresh data, just submit the last sample ID that you already have as start position. The ID is unique and auto incrementing.
Then the server returns up to 5000 new records with the first record starting at start_position + 1.
Now after you import the data, just repeat the process (remember that during import, your last ID went up), and eventually you'll get the entire trickle database (remember, it only contains last 31-32 days).
I've got a sample PHP script which does that (it's really simple and works just fine), I think I'll post it later after some cleaning up.
If you need daily data, I think it's better to synch your database with ours and then do the queries locally, as it'll be much faster.
Also, I'm dumping the entire trickles archive, I'll upload it to dropbox and post the URL later today. |
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
https://dl.dropbox.com/u/1912652/trickles_lite_08_14.rar
https://dl.dropbox.com/u/1912652/trickles_lite_archive_08_14.rar
____________
|
|
|
|
thank you very much TJM. We are currently running some analyses of the data and will see about the results. The method to download the data will also be discussed. :-) |
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
Please remember, that this is the full archive and it contains all our data, including our initial tests, some of which were done with a square wave generator connected to the sensor input.
Also make sure you exclude the samples marked with experiment flag from any calculations.
|
|
|
|
thanks TJM,
I forwarded this info to my colleague. :-)
Second, what do you think about adding a new parameter marking whether the sensor is indoor or outdoors?
I am not sure whether the sensor is able to be placed outdoors without significant modifications so maybe such parameter would be useless. |
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
Here's my script that I use to pull the new data, it's not commented at all but should be easy to understand anyway.
For performance reasons it's set to stop after 20 iterations (so each run it pulls up to 100k records) or when there's no more data available.
header('content-type: text/plain');
$rows_inserted = 1; //zeby wystartowac petle
$num_updates = 0;
while ($rows_inserted > 0 and $num_updates < 20) {
$result = mysql_query("SELECT MAX(id) FROM trickles_lite") OR die(mysql_error());
if (mysql_num_rows($result) == 0) {
$max_id = 0;
}
else {
$wynik = mysql_fetch_row($result);
$max_id = $wynik[0];
}
echo "Last local trickle id: $max_id\n";
$content = file_get_contents("http://radioactiveathome.org/boinc/gettrickledata.php?start=$max_id");
$content_array = explode("\n",$content);
if ($content_array[0] == "NO DATA") die("Remote server returned empty document");
$first_line = explode(":",$content_array[0]);
$remote_min_trickle_id = $first_line[0];
$remote_max_trickle_id = $first_line[1];
$content_size = sizeof($content_array);
echo "$content_size lines of data\n";
echo "Remote data min id: $remote_min_trickle_id, max id: $remote_max_trickle_id\n";
$rows_inserted = 0;
for ($i=1;$i<$content_size;$i++) {
if ($content_array[$i] == "NO DATA") continue;
$line_content = explode(",",$content_array[$i]);
$tdata_id = $line_content[0];
$tdata_hostid = $line_content[1];
$tdata_pomiar_raw = $line_content[2];
$tdata_data = $line_content[3];
$tdata_loc1 = $line_content[4];
$tdata_loc2 = $line_content[5];
$tdata_czas_pomiaru = $line_content[6];
$tdata_typ_pomiaru = $line_content[7];
$tdata_eksperyment = $line_content[8];
$tdata_revision_int = $line_content[9];
$tdata_vid_pid_int = 0; //to be changed later
if ($tdata_id) {
mysql_query("INSERT INTO trickles_lite vALUES($tdata_id,$tdata_hostid, $tdata_pomiar_raw, '$tdata_data', $tdata_loc1, $tdata_loc2, $tdata_czas_pomiaru, '$tdata_typ_pomiaru', $tdata_eksperyment, $tdata_revision_int, $tdata_vid_pid_int)") OR die(mysql_error());
$rows_inserted++;
}
}
echo "$rows_inserted records written to the local db\n";
$num_updates++;
} //while
____________
|
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
Indoor/Outdoors preferences option and a database field would be rather easy to implement.
Other thing that may be added to the sample database is the geiger tube type, for now it's not really necessary as all the detectors use either SBM-20, STS-5 or compatible GM tubes, however it would be nice to support other tubes, some of which will have completely different characteristics. |
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
Here is a dump of our data archive, .csv files, separate file for each month, I'll try to update it monthly.
2011 as complete archive: https://dl.dropbox.com/u/1912652/radioactive_data/2011.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2011_06.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2011_07.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2011_08.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2011_09.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2011_10.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2011_11.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2011_12.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_01.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_02.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_03.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_04.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_05.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_06.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_07.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_08.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_09.7z
If I didn't mess up anything, the data format should be:
sample_id (unique), host_id, pulse_count, sample_time, datestamp (UTC), sample_type, loc1, loc2, sensor_revision |
|
|
|
Hello,
I need to recover the last 15 days from host ID 1787, because I've try to measure some bags of pellet ashes and I would like to do some calculations on the raw data.
And I've see that these file:
https://dl.dropbox.com/u/1912652/radioactive_data/2013_03.7z or
https://dl.dropbox.com/u/1912652/radioactive_data/2013_04.7z
do not exists.
Have you another site to download the raw data ?
Regards,
Ale
|
|
|
|
For everyone interested in getting the raw data returned from sensors, there is a simple script that allows synchronization with the database.
http://radioactiveathome.org/boinc/gettrickledata.php
http://radioactiveathome.org/boinc/gettrickledata.php?start=X&hostid=Y
With this link:
http://radioactiveathome.org/boinc/gettrickledata.php?start=0&hostid=1787
Actually I got 5000 lines but with date from 2013-03-10 to 2013-03-13:
135397046:151614761
135397046,1787,7,2013-03-10 23:46:29,44.423836,8.859959,0.700,n,0,593
..
..
136637016,1787,6,2013-03-13 22:49:39,44.423836,8.859959,0.701,n,0,593
136637017,1787,7,2013-03-13 22:50:21,44.423836,8.859959,0.701,n,0,593
not the last today (2013-03-18) samples. Why ?
The live database is limited up to 31 days back (it is cleared from time to time).
31 days = ( 31 * 24 * 3600 ) / 42 = about 63771 samples
how I can get more samples ?
|
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
You have to provide the "start" position, like this:
http://radioactiveathome.org/boinc/gettrickledata.php?start=136637017&hostid=1787
Just look at the last sample ID, if the data is still not what you want, use the last ID as start position and you'll get the next 5000 records. It's not very good for manual work, but works very well with scripts.
|
|
|
|
Could you please describe the data row format ones again? The image describing row format is unavailable. |
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
The first line is the min:max sample id currently available in the 'live' database.
The data format is:
trickle_id(unique), host_id, counts, datetime, location_1, location_2, sample_time (minutes), sample type, experiment mode (0=off, 1=on), sensor firmware revision (it's bcd encoded then converted to integer for example 574 = 0252 hex = v2.52).
Probably today I'll add another field (at the end) with sensor hardware id.
|
|
|
|
Hi,
As I see there are only some of the dumps available (the last dump is 2012_09.7z).
Could you please share the data from October 2012 till June 2013 inclusively. I'll try to build some time series models for the data.
What about creating torrent files for the old dumps so the people could download old data to build their local copies of the database? And then sync up with your database using 'live' data access.
It will free your dropbox space.
We can use public free torrent trackers like openbittorrent.com or publicbt.com
We can find some volunteers to seed the data all the time.
E.g. I can seed 1 or 2 years of data archive on my always online PC. |
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
I'll reupload the archives during the weekend, I think I'll also set up an automated dump every month.
|
|
|
|
I've seen some records with S sample type like
169943792,10000001,105,2013-07-18 04:46:17,0.000000,0.000000,5.000,S,1,0
TJM, Could you tell us what it is? |
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
'S' flag is for the standalone version of the sensor which communicates directly with the server without BOINC or even PC.
For now just ignore any 'S' records as the prototype is in early beta stage. |
|
|
|
I can't get my raw data per script any longer
I ran a php-script to get the raw data for my machine, and since 5th November I can't get them any longer. Here's the error I receive:
(( Warning:
file(http://radioactiveathome.org/boinc/gettrickledata.php?start=180072691&hostid=1722)
[function.file]: failed to open stream: HTTP request failed!
HTTP/1.1 403 Forbidden ))
If I go there by browser, I still get them, but this doesn't work any more.
Has there been some changes to the database recently, or is there something else wrong or broken?
|
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
I know what's going on and I'll fix the problem later today.
The server blocks all requests to /boinc/ if there is no user agent set.
|
|
|
|
Thank you |
|
|
TJMProject administrator Project developer Project tester Send message
Joined: 16 Apr 11 Posts: 291 Credit: 1,381,991 RAC: 67
|
Fixed.
|
|
|
|
Hi,
i've found a bug or feature i don't understand.
http://radioactiveathome.org/boinc/gettrickledata.php?start=201493843
12th line:
201493854,11135,-156,2014-07-01 07:45:33,48.498417,7.944239,4.009,n,0,769
----------------^
Is this a uint/int bug or are negative values handled in a different way.
Greetings,
jtf |
|
|
|
Hello
There is a problem with the file gettrickledata.php
the problem is Unable to connect to MySQL
|
|
|
|
Here's my script that I use to pull the new data, it's not commented at all but should be easy to understand anyway.
For performance reasons it's set to stop after 20 iterations (so each run it pulls up to 100k records) or when there's no more data available.
header('content-type: text/plain');
$rows_inserted = 1; //zeby wystartowac petle
$num_updates = 0;
while ($rows_inserted > 0 and $num_updates < 20) {
$result = mysql_query("SELECT MAX(id) FROM trickles_lite") OR die(mysql_error());
if (mysql_num_rows($result) == 0) {
$max_id = 0;
}
else {
$wynik = mysql_fetch_row($result);
$max_id = $wynik[0];
}
echo "Last local trickle id: $max_id\n";
$content = file_get_contents("http://radioactiveathome.org/boinc/gettrickledata.php?start=$max_id");
$content_array = explode("\n",$content);
if ($content_array[0] == "NO DATA") die("Remote server returned empty document");
$first_line = explode(":",$content_array[0]);
$remote_min_trickle_id = $first_line[0];
$remote_max_trickle_id = $first_line[1];
$content_size = sizeof($content_array);
echo "$content_size lines of data\n";
echo "Remote data min id: $remote_min_trickle_id, max id: $remote_max_trickle_id\n";
$rows_inserted = 0;
for ($i=1;$i<$content_size;$i++) {
if ($content_array[$i] == "NO DATA") continue;
$line_content = explode(",",$content_array[$i]);
$tdata_id = $line_content[0];
$tdata_hostid = $line_content[1];
$tdata_pomiar_raw = $line_content[2];
$tdata_data = $line_content[3];
$tdata_loc1 = $line_content[4];
$tdata_loc2 = $line_content[5];
$tdata_czas_pomiaru = $line_content[6];
$tdata_typ_pomiaru = $line_content[7];
$tdata_eksperyment = $line_content[8];
$tdata_revision_int = $line_content[9];
$tdata_vid_pid_int = 0; //to be changed later
if ($tdata_id) {
mysql_query("INSERT INTO trickles_lite vALUES($tdata_id,$tdata_hostid, $tdata_pomiar_raw, '$tdata_data', $tdata_loc1, $tdata_loc2, $tdata_czas_pomiaru, '$tdata_typ_pomiaru', $tdata_eksperyment, $tdata_revision_int, $tdata_vid_pid_int)") OR die(mysql_error());
$rows_inserted++;
}
}
echo "$rows_inserted records written to the local db\n";
$num_updates++;
} //while
Hi TJM,
could you post the complette php script?
And it is possible to tell us what should we do when i would like inport in to the sql database.
thanks
Werner |
|
|
krzyszpProject administrator Project developer Project tester Project scientist
Send message
Joined: 16 Apr 11 Posts: 383 Credit: 785,913 RAC: 144
|
This is complete script ;)
This is query which writes data to database:
mysql_query("INSERT INTO trickles_lite VALUES($tdata_id,$tdata_hostid, $tdata_pomiar_raw, '$tdata_data', $tdata_loc1, $tdata_loc2, $tdata_czas_pomiaru, '$tdata_typ_pomiaru', $tdata_eksperyment, $tdata_revision_int, $tdata_vid_pid_int)") OR die(mysql_error());
____________
Regards,
Krzysztof 'krzyszp' Piszczek
Android Radioactive@Home Map
Android Radioactive@Home Map - donated
My Workplace |
|
|
|
Hello,
i get an error when i use the Query on my MySQL DB.
I using the SQL Software-Version: 5.1.61 - Source distribution Protokoll-Version: 10
After start the query i get following error.
#1064 - You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'header('content-type: text/plain')' at line 1
Using following Script with my stationsID.
header('content-type: text/plain');
$rows_inserted = 1; //zeby wystartowac petle
$num_updates = 0;
while ($rows_inserted > 0 and $num_updates < 20) {
$result = mysql_query("SELECT MAX(id) FROM trickles_lite") OR die(mysql_error());
if (mysql_num_rows($result) == 0) {
$max_id = 0;
}
else {
$wynik = mysql_fetch_row($result);
$max_id = $wynik[0];
}
echo "Last local trickle id: $max_id\n";
$content = file_get_contents("http://radioactiveathome.org/boinc/gettrickledata.php?start=0&hostid=12104");
$content_array = explode("\n",$content);
if ($content_array[0] == "NO DATA") die("Remote server returned empty document");
$first_line = explode(":",$content_array[0]);
$remote_min_trickle_id = $first_line[0];
$remote_max_trickle_id = $first_line[1];
$content_size = sizeof($content_array);
echo "$content_size lines of data\n";
echo "Remote data min id: $remote_min_trickle_id, max id: $remote_max_trickle_id\n";
$rows_inserted = 0;
for ($i=1;$i<$content_size;$i++) {
if ($content_array[$i] == "NO DATA") continue;
$line_content = explode(",",$content_array[$i]);
$tdata_id = $line_content[0];
$tdata_hostid = $line_content[1];
$tdata_pomiar_raw = $line_content[2];
$tdata_data = $line_content[3];
$tdata_loc1 = $line_content[4];
$tdata_loc2 = $line_content[5];
$tdata_czas_pomiaru = $line_content[6];
$tdata_typ_pomiaru = $line_content[7];
$tdata_eksperyment = $line_content[8];
$tdata_revision_int = $line_content[9];
$tdata_vid_pid_int = 0; //to be changed later
if ($tdata_id) {
mysql_query("INSERT INTO trickles_lite vALUES($tdata_id,$tdata_hostid, $tdata_pomiar_raw, '$tdata_data', $tdata_loc1, $tdata_loc2, $tdata_czas_pomiaru, '$tdata_typ_pomiaru', $tdata_eksperyment, $tdata_revision_int, $tdata_vid_pid_int)") OR die(mysql_error());
$rows_inserted++;
}
}
echo "$rows_inserted records written to the local db\n";
$num_updates++;
} //while |
|
|
|
Hi,
It seems that raw data export has trouble now.
I see some PHP code instead of the data when I access http://radioactiveathome.org/boinc/gettrickledata.php
Could someone check please? |
|
|