Raw data export


Advanced search

Message boards : Number crunching : Raw data export

Author Message
Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 574 - Posted: 2 Dec 2011 | 9:39:31 UTC
Last modified: 8 Sep 2013 | 16:36:30 UTC

For everyone interested in getting the raw data returned from sensors, there is a simple script that allows synchronization with the database.

http://radioactiveathome.org/boinc/gettrickledata.php
http://radioactiveathome.org/boinc/gettrickledata.php?start=X&hostid=Y

The first line shows min:max sample ID currently available in the database, then up to 5000 lines of samples are returned.
"hostid" is optional and limits returned data to single host.
"start" skips data until last position you already have (or 0 to read from the beginning), the first returned sample ID is start+1 (or the minimum id available in the DB).

Row data format:



The live database is limited up to 31 days back (it is cleared from time to time).
There are currently 3 types of samples: "f" - first one read from sensor at the app start, "n" - normal sample taken when app is running, "r" - sample taken when the app was resumed/restarted. Both "f" and "r" may contain "long" samples taken for example when the PC was left in standby and the sensor was running.

For older data take a look here:
http://radioactiveathome.org/boinc/download/data/

Dagorath
Avatar
Send message
Joined: 4 Jul 11
Posts: 151
Credit: 42,738
RAC: 0

Message 700 - Posted: 15 Jan 2012 | 22:05:28 UTC - in response to Message 574.

Is there a formula for calculating uSv/h from the pulse count and sample time values?

____________

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 716 - Posted: 25 Jan 2012 | 15:03:21 UTC

I believe it's (number_of_pulses/time_in_minutes)/171.232876

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 755 - Posted: 19 Feb 2012 | 10:46:44 UTC - in response to Message 716.

The script was updated, the last digit is the "experiment" flag which is set to 1 if the user marks his readings as test results (testing hardware/software/tubes etc).

____________

Dagorath
Avatar
Send message
Joined: 4 Jul 11
Posts: 151
Credit: 42,738
RAC: 0

Message 854 - Posted: 12 Mar 2012 | 6:22:07 UTC - in response to Message 755.

There appears to be a new field after the experimental flag.

____________

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 855 - Posted: 12 Mar 2012 | 8:55:52 UTC - in response to Message 854.

Convert the integer to hexadecimal to get two bytes with BCD-encoded sensor hardware/software version.
It's not fully supported yet, as it requires app update. Windows version 1.56 takes care of it, while Linux version will come later today.

Dagorath
Avatar
Send message
Joined: 4 Jul 11
Posts: 151
Credit: 42,738
RAC: 0

Message 861 - Posted: 15 Mar 2012 | 8:28:03 UTC - in response to Message 855.

Thanks, TJM.

____________

Profile WetterYbbs
Send message
Joined: 15 Mar 12
Posts: 9
Credit: 349,735
RAC: 0

Message 913 - Posted: 18 Mar 2012 | 13:45:05 UTC

I have a problem, there are no more data is written to the last entry is 03/17/2012 23:52 why
look here http://radioactiveathome.org/boinc/gettrickledata.php?start=X&hostid=1722

lg,
Wolfgang

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 914 - Posted: 18 Mar 2012 | 13:54:31 UTC - in response to Message 913.

Look at the link you provided, you have left "X" where the script expects starting position. Without knowing the position it returns first 5000 records.

Profile WetterYbbs
Send message
Joined: 15 Mar 12
Posts: 9
Credit: 349,735
RAC: 0

Message 915 - Posted: 18 Mar 2012 | 17:14:03 UTC
Last modified: 18 Mar 2012 | 17:14:33 UTC

Sorry, my mistake Thanks for the answer TJM

lg,
Wolfgang
____________

Profile jhelebrant
Avatar
Send message
Joined: 30 Jul 12
Posts: 27
Credit: 1,521
RAC: 0
Message 1293 - Posted: 13 Aug 2012 | 13:41:43 UTC

hi guys,
many thanks for all this info.

However, is there any simple trick (script etc.) to download the whole database or all data for single station without manual downloading or generating plenty of download links?

We would like to perform some analyses of long term data (1 month, first only for 1-2 stations) and later maybe more.

thanks much

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 1295 - Posted: 13 Aug 2012 | 14:14:15 UTC - in response to Message 1293.

I can provide full dump of the samples database, but it's so huge, that it can't be exported regularly (takes a lot of time and bandwidth).
The best way to keep up with our data is to import the dump and then write a script that pulls fresh data from the server every now and then, it doesn't stress the server and keeps the traffic low, as only new data is downloaded on each run.

Profile jhelebrant
Avatar
Send message
Joined: 30 Jul 12
Posts: 27
Credit: 1,521
RAC: 0
Message 1296 - Posted: 14 Aug 2012 | 11:36:43 UTC - in response to Message 1295.
Last modified: 14 Aug 2012 | 11:37:36 UTC

Thanks for reply TJM.
We are not going to download full dump regularly, but we would need it as start point.

I would appreciate any advice in this case.
For example if I use this link:
http://radioactiveathome.org/boinc/gettrickledata.php?start=X&hostid=Y
without any parameter set,
I get output for 48763789 to 48770832, while there is written that the database contains data 48763789:60449876

If I modify the link to:
http://radioactiveathome.org/boinc/gettrickledata.php?start=60449876&hostid=Y

I get the last portion of the data.
However, if I am right, the Sample ID (starts now at 48763789) changes, so I cannot use it as fixed value in a script.

I know, that I can jump to next data page by using the last sample id as the X parameter in the link.

Second, it would be necessary - if we consider daily database update - to somehow set the script to only load pages for the particular day.

Is there any additional parameter which could be used for this task or do I have to somehow calculate/guess the approximate number of data pages, which should be checked?

thanks

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 1297 - Posted: 14 Aug 2012 | 14:31:40 UTC - in response to Message 1296.
Last modified: 14 Aug 2012 | 14:32:49 UTC

To pull fresh data, just submit the last sample ID that you already have as start position. The ID is unique and auto incrementing.
Then the server returns up to 5000 new records with the first record starting at start_position + 1.
Now after you import the data, just repeat the process (remember that during import, your last ID went up), and eventually you'll get the entire trickle database (remember, it only contains last 31-32 days).
I've got a sample PHP script which does that (it's really simple and works just fine), I think I'll post it later after some cleaning up.
If you need daily data, I think it's better to synch your database with ours and then do the queries locally, as it'll be much faster.

Also, I'm dumping the entire trickles archive, I'll upload it to dropbox and post the URL later today.

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 1300 - Posted: 15 Aug 2012 | 16:10:43 UTC - in response to Message 1297.

https://dl.dropbox.com/u/1912652/trickles_lite_08_14.rar
https://dl.dropbox.com/u/1912652/trickles_lite_archive_08_14.rar

____________

Profile jhelebrant
Avatar
Send message
Joined: 30 Jul 12
Posts: 27
Credit: 1,521
RAC: 0
Message 1303 - Posted: 17 Aug 2012 | 14:05:37 UTC - in response to Message 1300.

thank you very much TJM. We are currently running some analyses of the data and will see about the results. The method to download the data will also be discussed. :-)

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 1313 - Posted: 19 Aug 2012 | 21:11:00 UTC - in response to Message 1303.

Please remember, that this is the full archive and it contains all our data, including our initial tests, some of which were done with a square wave generator connected to the sensor input.
Also make sure you exclude the samples marked with experiment flag from any calculations.

Profile jhelebrant
Avatar
Send message
Joined: 30 Jul 12
Posts: 27
Credit: 1,521
RAC: 0
Message 1347 - Posted: 31 Aug 2012 | 7:57:57 UTC - in response to Message 1313.

thanks TJM,
I forwarded this info to my colleague. :-)

Second, what do you think about adding a new parameter marking whether the sensor is indoor or outdoors?

I am not sure whether the sensor is able to be placed outdoors without significant modifications so maybe such parameter would be useless.

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 1349 - Posted: 31 Aug 2012 | 10:48:09 UTC - in response to Message 1347.

Here's my script that I use to pull the new data, it's not commented at all but should be easy to understand anyway.
For performance reasons it's set to stop after 20 iterations (so each run it pulls up to 100k records) or when there's no more data available.


header('content-type: text/plain');
$rows_inserted = 1; //zeby wystartowac petle
$num_updates = 0;
while ($rows_inserted > 0 and $num_updates < 20) {
$result = mysql_query("SELECT MAX(id) FROM trickles_lite") OR die(mysql_error());
if (mysql_num_rows($result) == 0) {
$max_id = 0;
}
else {
$wynik = mysql_fetch_row($result);
$max_id = $wynik[0];
}
echo "Last local trickle id: $max_id\n";

$content = file_get_contents("http://radioactiveathome.org/boinc/gettrickledata.php?start=$max_id");
$content_array = explode("\n",$content);
if ($content_array[0] == "NO DATA") die("Remote server returned empty document");
$first_line = explode(":",$content_array[0]);
$remote_min_trickle_id = $first_line[0];
$remote_max_trickle_id = $first_line[1];
$content_size = sizeof($content_array);
echo "$content_size lines of data\n";
echo "Remote data min id: $remote_min_trickle_id, max id: $remote_max_trickle_id\n";
$rows_inserted = 0;
for ($i=1;$i<$content_size;$i++) {
if ($content_array[$i] == "NO DATA") continue;
$line_content = explode(",",$content_array[$i]);
$tdata_id = $line_content[0];
$tdata_hostid = $line_content[1];
$tdata_pomiar_raw = $line_content[2];
$tdata_data = $line_content[3];
$tdata_loc1 = $line_content[4];
$tdata_loc2 = $line_content[5];
$tdata_czas_pomiaru = $line_content[6];
$tdata_typ_pomiaru = $line_content[7];
$tdata_eksperyment = $line_content[8];
$tdata_revision_int = $line_content[9];
$tdata_vid_pid_int = 0; //to be changed later
if ($tdata_id) {
mysql_query("INSERT INTO trickles_lite vALUES($tdata_id,$tdata_hostid, $tdata_pomiar_raw, '$tdata_data', $tdata_loc1, $tdata_loc2, $tdata_czas_pomiaru, '$tdata_typ_pomiaru', $tdata_eksperyment, $tdata_revision_int, $tdata_vid_pid_int)") OR die(mysql_error());
$rows_inserted++;
}
}
echo "$rows_inserted records written to the local db\n";
$num_updates++;
} //while

____________

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 1351 - Posted: 31 Aug 2012 | 21:24:48 UTC - in response to Message 1349.
Last modified: 31 Aug 2012 | 21:25:18 UTC

Indoor/Outdoors preferences option and a database field would be rather easy to implement.
Other thing that may be added to the sample database is the geiger tube type, for now it's not really necessary as all the detectors use either SBM-20, STS-5 or compatible GM tubes, however it would be nice to support other tubes, some of which will have completely different characteristics.

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 1386 - Posted: 11 Oct 2012 | 19:22:29 UTC
Last modified: 11 Oct 2012 | 19:23:14 UTC

Here is a dump of our data archive, .csv files, separate file for each month, I'll try to update it monthly.

2011 as complete archive: https://dl.dropbox.com/u/1912652/radioactive_data/2011.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2011_06.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2011_07.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2011_08.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2011_09.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2011_10.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2011_11.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2011_12.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_01.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_02.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_03.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_04.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_05.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_06.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_07.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_08.7z
https://dl.dropbox.com/u/1912652/radioactive_data/2012_09.7z

If I didn't mess up anything, the data format should be:

sample_id (unique), host_id, pulse_count, sample_time, datestamp (UTC), sample_type, loc1, loc2, sensor_revision

Alessandro Freda
Send message
Joined: 17 Aug 11
Posts: 37
Credit: 1,697,843
RAC: 148

Message 1696 - Posted: 18 Apr 2013 | 12:17:30 UTC - in response to Message 1386.

Hello,

I need to recover the last 15 days from host ID 1787, because I've try to measure some bags of pellet ashes and I would like to do some calculations on the raw data.

And I've see that these file:

https://dl.dropbox.com/u/1912652/radioactive_data/2013_03.7z or
https://dl.dropbox.com/u/1912652/radioactive_data/2013_04.7z

do not exists.
Have you another site to download the raw data ?

Regards,

Ale

Alessandro Freda
Send message
Joined: 17 Aug 11
Posts: 37
Credit: 1,697,843
RAC: 148

Message 1697 - Posted: 18 Apr 2013 | 15:54:49 UTC - in response to Message 574.

For everyone interested in getting the raw data returned from sensors, there is a simple script that allows synchronization with the database.

http://radioactiveathome.org/boinc/gettrickledata.php
http://radioactiveathome.org/boinc/gettrickledata.php?start=X&hostid=Y

With this link:
http://radioactiveathome.org/boinc/gettrickledata.php?start=0&hostid=1787
Actually I got 5000 lines but with date from 2013-03-10 to 2013-03-13:

135397046:151614761
135397046,1787,7,2013-03-10 23:46:29,44.423836,8.859959,0.700,n,0,593
..
..
136637016,1787,6,2013-03-13 22:49:39,44.423836,8.859959,0.701,n,0,593
136637017,1787,7,2013-03-13 22:50:21,44.423836,8.859959,0.701,n,0,593

not the last today (2013-03-18) samples. Why ?

The live database is limited up to 31 days back (it is cleared from time to time).

31 days = ( 31 * 24 * 3600 ) / 42 = about 63771 samples
how I can get more samples ?

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 1699 - Posted: 18 Apr 2013 | 19:03:01 UTC - in response to Message 1697.

You have to provide the "start" position, like this:

http://radioactiveathome.org/boinc/gettrickledata.php?start=136637017&hostid=1787

Just look at the last sample ID, if the data is still not what you want, use the last ID as start position and you'll get the next 5000 records. It's not very good for manual work, but works very well with scripts.

Dmitry
Avatar
Send message
Joined: 27 Jul 12
Posts: 22
Credit: 271,107
RAC: 0

Message 1827 - Posted: 29 Jun 2013 | 6:59:15 UTC - in response to Message 574.

Could you please describe the data row format ones again? The image describing row format is unavailable.

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 1828 - Posted: 29 Jun 2013 | 7:50:46 UTC - in response to Message 1827.

The first line is the min:max sample id currently available in the 'live' database.

The data format is:
trickle_id(unique), host_id, counts, datetime, location_1, location_2, sample_time (minutes), sample type, experiment mode (0=off, 1=on), sensor firmware revision (it's bcd encoded then converted to integer for example 574 = 0252 hex = v2.52).

Probably today I'll add another field (at the end) with sensor hardware id.


Dmitry
Avatar
Send message
Joined: 27 Jul 12
Posts: 22
Credit: 271,107
RAC: 0

Message 1866 - Posted: 8 Jul 2013 | 17:09:19 UTC - in response to Message 1828.
Last modified: 8 Jul 2013 | 17:32:47 UTC

Hi,

As I see there are only some of the dumps available (the last dump is 2012_09.7z).

Could you please share the data from October 2012 till June 2013 inclusively. I'll try to build some time series models for the data.



What about creating torrent files for the old dumps so the people could download old data to build their local copies of the database? And then sync up with your database using 'live' data access.

It will free your dropbox space.

We can use public free torrent trackers like openbittorrent.com or publicbt.com

We can find some volunteers to seed the data all the time.
E.g. I can seed 1 or 2 years of data archive on my always online PC.

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 1906 - Posted: 18 Jul 2013 | 19:40:50 UTC - in response to Message 1866.

I'll reupload the archives during the weekend, I think I'll also set up an automated dump every month.

Dmitry
Avatar
Send message
Joined: 27 Jul 12
Posts: 22
Credit: 271,107
RAC: 0

Message 1916 - Posted: 22 Jul 2013 | 16:02:35 UTC
Last modified: 22 Jul 2013 | 16:12:35 UTC

I've seen some records with S sample type like

169943792,10000001,105,2013-07-18 04:46:17,0.000000,0.000000,5.000,S,1,0

TJM, Could you tell us what it is?

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 1917 - Posted: 22 Jul 2013 | 17:02:28 UTC - in response to Message 1916.

'S' flag is for the standalone version of the sensor which communicates directly with the server without BOINC or even PC.
For now just ignore any 'S' records as the prototype is in early beta stage.

Profile WetterYbbs
Send message
Joined: 15 Mar 12
Posts: 9
Credit: 349,735
RAC: 0

Message 2078 - Posted: 15 Nov 2013 | 13:56:03 UTC

I can't get my raw data per script any longer

I ran a php-script to get the raw data for my machine, and since 5th November I can't get them any longer. Here's the error I receive:
(( Warning:
file(http://radioactiveathome.org/boinc/gettrickledata.php?start=180072691&hostid=1722)
[function.file]: failed to open stream: HTTP request failed!
HTTP/1.1 403 Forbidden ))

If I go there by browser, I still get them, but this doesn't work any more.
Has there been some changes to the database recently, or is there something else wrong or broken?

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 2079 - Posted: 15 Nov 2013 | 14:53:24 UTC - in response to Message 2078.

I know what's going on and I'll fix the problem later today.
The server blocks all requests to /boinc/ if there is no user agent set.

Profile WetterYbbs
Send message
Joined: 15 Mar 12
Posts: 9
Credit: 349,735
RAC: 0

Message 2080 - Posted: 15 Nov 2013 | 19:13:39 UTC

Thank you

Profile TJM
Project administrator
Project developer
Project tester
Send message
Joined: 16 Apr 11
Posts: 291
Credit: 1,368,752
RAC: 58

Message 2081 - Posted: 15 Nov 2013 | 22:37:17 UTC - in response to Message 2080.

Fixed.

Profile jtf
Send message
Joined: 17 Mar 14
Posts: 5
Credit: 60,775
RAC: 0

Message 2852 - Posted: 29 Jul 2014 | 22:16:58 UTC

Hi,

i've found a bug or feature i don't understand.

http://radioactiveathome.org/boinc/gettrickledata.php?start=201493843

12th line:


201493854,11135,-156,2014-07-01 07:45:33,48.498417,7.944239,4.009,n,0,769
----------------^

Is this a uint/int bug or are negative values handled in a different way.

Greetings,
jtf

Profile WetterYbbs
Send message
Joined: 15 Mar 12
Posts: 9
Credit: 349,735
RAC: 0

Message 2857 - Posted: 1 Aug 2014 | 8:52:55 UTC

Hello
There is a problem with the file gettrickledata.php
the problem is Unable to connect to MySQL

gismoat
Send message
Joined: 5 Apr 13
Posts: 27
Credit: 252,101
RAC: 0

Message 2936 - Posted: 2 Oct 2014 | 21:22:26 UTC - in response to Message 1349.

Here's my script that I use to pull the new data, it's not commented at all but should be easy to understand anyway.
For performance reasons it's set to stop after 20 iterations (so each run it pulls up to 100k records) or when there's no more data available.


header('content-type: text/plain');
$rows_inserted = 1; //zeby wystartowac petle
$num_updates = 0;
while ($rows_inserted > 0 and $num_updates < 20) {
$result = mysql_query("SELECT MAX(id) FROM trickles_lite") OR die(mysql_error());
if (mysql_num_rows($result) == 0) {
$max_id = 0;
}
else {
$wynik = mysql_fetch_row($result);
$max_id = $wynik[0];
}
echo "Last local trickle id: $max_id\n";

$content = file_get_contents("http://radioactiveathome.org/boinc/gettrickledata.php?start=$max_id");
$content_array = explode("\n",$content);
if ($content_array[0] == "NO DATA") die("Remote server returned empty document");
$first_line = explode(":",$content_array[0]);
$remote_min_trickle_id = $first_line[0];
$remote_max_trickle_id = $first_line[1];
$content_size = sizeof($content_array);
echo "$content_size lines of data\n";
echo "Remote data min id: $remote_min_trickle_id, max id: $remote_max_trickle_id\n";
$rows_inserted = 0;
for ($i=1;$i<$content_size;$i++) {
if ($content_array[$i] == "NO DATA") continue;
$line_content = explode(",",$content_array[$i]);
$tdata_id = $line_content[0];
$tdata_hostid = $line_content[1];
$tdata_pomiar_raw = $line_content[2];
$tdata_data = $line_content[3];
$tdata_loc1 = $line_content[4];
$tdata_loc2 = $line_content[5];
$tdata_czas_pomiaru = $line_content[6];
$tdata_typ_pomiaru = $line_content[7];
$tdata_eksperyment = $line_content[8];
$tdata_revision_int = $line_content[9];
$tdata_vid_pid_int = 0; //to be changed later
if ($tdata_id) {
mysql_query("INSERT INTO trickles_lite vALUES($tdata_id,$tdata_hostid, $tdata_pomiar_raw, '$tdata_data', $tdata_loc1, $tdata_loc2, $tdata_czas_pomiaru, '$tdata_typ_pomiaru', $tdata_eksperyment, $tdata_revision_int, $tdata_vid_pid_int)") OR die(mysql_error());
$rows_inserted++;
}
}
echo "$rows_inserted records written to the local db\n";
$num_updates++;
} //while


Hi TJM,
could you post the complette php script?
And it is possible to tell us what should we do when i would like inport in to the sql database.

thanks
Werner

Profile krzyszp
Project administrator
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 16 Apr 11
Posts: 383
Credit: 754,646
RAC: 85

Message 2937 - Posted: 3 Oct 2014 | 9:08:15 UTC - in response to Message 2936.

This is complete script ;)

This is query which writes data to database:

mysql_query("INSERT INTO trickles_lite VALUES($tdata_id,$tdata_hostid, $tdata_pomiar_raw, '$tdata_data', $tdata_loc1, $tdata_loc2, $tdata_czas_pomiaru, '$tdata_typ_pomiaru', $tdata_eksperyment, $tdata_revision_int, $tdata_vid_pid_int)") OR die(mysql_error());

____________
Regards,
Krzysztof 'krzyszp' Piszczek
Android Radioactive@Home Map
Android Radioactive@Home Map - donated
My Workplace

gismoat
Send message
Joined: 5 Apr 13
Posts: 27
Credit: 252,101
RAC: 0

Message 2939 - Posted: 4 Oct 2014 | 18:33:59 UTC
Last modified: 4 Oct 2014 | 18:35:34 UTC

Hello,

i get an error when i use the Query on my MySQL DB.
I using the SQL Software-Version: 5.1.61 - Source distribution Protokoll-Version: 10

After start the query i get following error.

#1064 - You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'header('content-type: text/plain')' at line 1


Using following Script with my stationsID.

header('content-type: text/plain');
$rows_inserted = 1; //zeby wystartowac petle
$num_updates = 0;
while ($rows_inserted > 0 and $num_updates < 20) {
$result = mysql_query("SELECT MAX(id) FROM trickles_lite") OR die(mysql_error());
if (mysql_num_rows($result) == 0) {
$max_id = 0;
}
else {
$wynik = mysql_fetch_row($result);
$max_id = $wynik[0];
}
echo "Last local trickle id: $max_id\n";

$content = file_get_contents("http://radioactiveathome.org/boinc/gettrickledata.php?start=0&hostid=12104");
$content_array = explode("\n",$content);
if ($content_array[0] == "NO DATA") die("Remote server returned empty document");
$first_line = explode(":",$content_array[0]);
$remote_min_trickle_id = $first_line[0];
$remote_max_trickle_id = $first_line[1];
$content_size = sizeof($content_array);
echo "$content_size lines of data\n";
echo "Remote data min id: $remote_min_trickle_id, max id: $remote_max_trickle_id\n";
$rows_inserted = 0;
for ($i=1;$i<$content_size;$i++) {
if ($content_array[$i] == "NO DATA") continue;
$line_content = explode(",",$content_array[$i]);
$tdata_id = $line_content[0];
$tdata_hostid = $line_content[1];
$tdata_pomiar_raw = $line_content[2];
$tdata_data = $line_content[3];
$tdata_loc1 = $line_content[4];
$tdata_loc2 = $line_content[5];
$tdata_czas_pomiaru = $line_content[6];
$tdata_typ_pomiaru = $line_content[7];
$tdata_eksperyment = $line_content[8];
$tdata_revision_int = $line_content[9];
$tdata_vid_pid_int = 0; //to be changed later
if ($tdata_id) {
mysql_query("INSERT INTO trickles_lite vALUES($tdata_id,$tdata_hostid, $tdata_pomiar_raw, '$tdata_data', $tdata_loc1, $tdata_loc2, $tdata_czas_pomiaru, '$tdata_typ_pomiaru', $tdata_eksperyment, $tdata_revision_int, $tdata_vid_pid_int)") OR die(mysql_error());
$rows_inserted++;
}
}
echo "$rows_inserted records written to the local db\n";
$num_updates++;
} //while

Dmitry
Avatar
Send message
Joined: 27 Jul 12
Posts: 22
Credit: 271,107
RAC: 0

Message 3487 - Posted: 28 Sep 2016 | 13:23:43 UTC

Hi,

It seems that raw data export has trouble now.

I see some PHP code instead of the data when I access http://radioactiveathome.org/boinc/gettrickledata.php

Could someone check please?

Post to thread

Message boards : Number crunching : Raw data export


Main page · Your account · Message boards


Copyright © 2024 BOINC@Poland | Open Science for the future