Server change requirments


Advanced search

Message boards : News : Server change requirments

Author Message
Profile krzyszp
Project administrator
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 16 Apr 11
Posts: 368
Credit: 453,845
RAC: 141

Message 768 - Posted: 25 Feb 2012 | 15:04:22 UTC
Last modified: 26 Feb 2012 | 15:54:40 UTC

Radioactive@Home is currently hosted on VPS server.
This server is not fast and reliable enough for expected quantity of detectors (soon more than 300) and large data expected and we need to move to dedicated server.

Unfortunately, cost of dedicated server is too high to cover it from our pockets so we would like to get some help from you by PayPal donations.

As soon as possible we would like to add donor badge for people who help us in cover project costs.

Many thanks for persons, who already donate our project and for persons who do it in future :)
____________
Regards,
Krzysztof 'krzyszp' Piszczek
Android Radioactive@Home Map
Android Radioactive@Home Map - donated
My Workplace

Profile Saenger
Send message
Joined: 16 Jun 11
Posts: 134
Credit: 368,074
RAC: 158

Message 769 - Posted: 26 Feb 2012 | 11:47:21 UTC
Last modified: 26 Feb 2012 | 11:57:36 UTC

Radioactive@Home ist aktuell auf einem VPS-Server (Virtueller private Server) gehostet.

Dieser Server ist nicht schnell und zuverlässig genug für die erwartete Anzahl Detektoren (bald über 300) und Datenmengen, wir müssen auf einen dedizierten Server umziehen.

Unglücklicherweise sind die Kosten für einen dedizierten Server zu groß, um sie aus unseren eigenen Mitteln zu bezahlen, wir würden daher gerne Hilfe von Euch in Form von Spenden bekommen.

Sobald möglich werden wir ein Spendenabzeichen für die Leute, die uns bei der Kostenübernahme geholfen haben, einführen.

Vielen Dank an die Leute, die uns bereits etwas gespendet haben, und an alle, die dies in Zukunft tun werden :)

Edith sagt übrigens:
Den Spendenknopf als solchen gibt es schon, auf der Hauptseite unter PayPal-Donations, nur das Abzeichen im Konto für die Spender, also das BOINC-Krönchen o.ä. fehlt noch.
____________
Gruesse vom Saenger

Dagorath
Avatar
Send message
Joined: 4 Jul 11
Posts: 151
Credit: 42,738
RAC: 0

Message 771 - Posted: 26 Feb 2012 | 21:00:38 UTC - in response to Message 768.
Last modified: 26 Feb 2012 | 21:15:58 UTC

How much money do you need? Are you going to buy a new server or rent from a hosting service? If you will rent then how much will the dedicated server cost per month?

I think if you explain the costs you will receive more donations. I can donate but first I need to know how much you need.

Also, my ISP is very generous and involved in community projects. If you want I can talk to them to see if they can donate bandwidth to your project since it is a non-profit community project. If they are willing to donate bandwidth then I can host data here on one of my Linux machines. I can give you whatever privileges you need to setup and administer the server yourself and provide dedicated RAID storage and daily backup to DVD.
____________

Profile krzyszp
Project administrator
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 16 Apr 11
Posts: 368
Credit: 453,845
RAC: 141

Message 772 - Posted: 26 Feb 2012 | 21:12:55 UTC - in response to Message 771.

Because some people already (positive) answer for our request, I have rent dedicated server already. Now I'm waiting for run it by hosting company.
This is Core i5-2400 (4x3.1GHz, HT), 8GB RAM, 60GB SSD server, cost about 65 Euro for month.
We have now collected money for about three months.

Also as my part in cost participation, I rent yesterday new VPS server (14 Euro monthly) to create files and SVN repository for project purposes.

During next week I would like make some tests (speed, reliability) on new server than (if everything be ok) move Radioactive@Home server to new machine.

I know, that's not a really big money, but all costs together are to high for my pocket now...
____________
Regards,
Krzysztof 'krzyszp' Piszczek
Android Radioactive@Home Map
Android Radioactive@Home Map - donated
My Workplace

Profile krzyszp
Project administrator
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 16 Apr 11
Posts: 368
Credit: 453,845
RAC: 141

Message 773 - Posted: 26 Feb 2012 | 21:33:17 UTC - in response to Message 771.


Also, my ISP is very generous and involved in community projects. If you want I can talk to them to see if they can donate bandwidth to your project since it is a non-profit community project. If they are willing to donate bandwidth then I can host data here on one of my Linux machines. I can give you whatever privileges you need to setup and administer the server yourself and provide dedicated RAID storage and daily backup to DVD.

That's interesting!

Maybe we discuss about storing public available copy of results database on its space?
How fast connection is available?

Can you send me some more info by PM, please?

____________
Regards,
Krzysztof 'krzyszp' Piszczek
Android Radioactive@Home Map
Android Radioactive@Home Map - donated
My Workplace

FogBRD
Send message
Joined: 16 Feb 12
Posts: 22
Credit: 6,008
RAC: 0
Message 775 - Posted: 27 Feb 2012 | 6:38:30 UTC - in response to Message 773.

I can not believe that this project needs such high demands for the equipment.
65Euro is for many of the salary in my country. Maybe we should review them?

Process the data from the 3000 pieces of detectors is only 300 megabytes per hour.
It is much more a problem is to keep public data archive. But that's project of distributed computing. I think every participant Projects are not difficult to distinguish for example 10GB of space on your disk dimensionless. So you get a large-scale data storage without the risk of losing it. The server used only for the current task for a month.
____________

Profile krzyszp
Project administrator
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 16 Apr 11
Posts: 368
Credit: 453,845
RAC: 141

Message 776 - Posted: 27 Feb 2012 | 12:34:50 UTC - in response to Message 775.

I can not believe that this project needs such high demands for the equipment.
65Euro is for many of the salary in my country. Maybe we should review them?

Process the data from the 3000 pieces of detectors is only 300 megabytes per hour.
It is much more a problem is to keep public data archive. But that's project of distributed computing. I think every participant Projects are not difficult to distinguish for example 10GB of space on your disk dimensionless. So you get a large-scale data storage without the risk of losing it. The server used only for the current task for a month.

It's not that simply...

Current database of results for only 60 detectors is bigger than 1GiB of data. This data are constantly updated and processed in system.
Additionally, MySQL all the time generate tasks, BOINC server all the time communicate with client machines, assign points, export data, generate tasks etc. Most task must be done in real time, which require... very quick access to HDD and memory.
Current VPS system is fast enough for 60 detectors but sometimes load on it growth dramatically and server lost communication with clients.
Also VPS as virtual system have very slow access to hard drive, depend on other systems on server.

I don't see any way to store data on participants computers as this solution is not reliable enough and don't give possibility to display data in real time on our web server.

Also we share results database for everybody and everybody can download it - this generate 1GiB of traffic each time when somebody do it... for now.
____________
Regards,
Krzysztof 'krzyszp' Piszczek
Android Radioactive@Home Map
Android Radioactive@Home Map - donated
My Workplace

FogBRD
Send message
Joined: 16 Feb 12
Posts: 22
Credit: 6,008
RAC: 0
Message 778 - Posted: 27 Feb 2012 | 14:30:49 UTC - in response to Message 776.

Oh. Well you have come up with a headache from scratch.
Too shy to ask - how much you know about radiation contamination / exposure?

What do you store raw data has no meaning. I'll try to explain.
We have. sensor installed in humans. Geiger is on the street (the house - it makes no sense).
sensor is connected to a computer and transmits data every 40 second measurements of the server. (if I understood correctly).
There is a notion - the relevance of the data. The question of how relevant the data on measurements of 40 seconds? Well, most people can point to the fact that "passed by a radioactive object", a word about anything. Other radioactive and techno-genetic accident gives a much longer background of countable hours or more. The minimum interval measurement in radiation reconnaissance accepted value of 30 minutes (if I remember correctly). Under no sense for a good reason.
Therefore, information on the map and in the base of the 40 second metering simply superfluous and irrelevant. So why such information is not stored. All you have nada to do an average of 30 minute interval and store the average value for these 30 minutes. Your base is reduced by 45 times smaller.
Anyway 40 seconds to measure the background gives a very large error (about 60%) for the SBM-20 counter (CTC-5), it is necessary to increase the time (or as I said average), and then the error will be approximately 20-30%.
And the graphics and data for people to become more clear and evident. You will then at least it is clear that they are for the "Saw" is shown?

What convinced you?
____________

Profile krzyszp
Project administrator
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 16 Apr 11
Posts: 368
Credit: 453,845
RAC: 141

Message 779 - Posted: 27 Feb 2012 | 14:43:00 UTC - in response to Message 778.

Measurements are done constantly and we store all data.
Every 35.04s is generated info for displaying on detector screen. For map we display last 1000 measurements, so everybody can recalculate it to get average for any period of time.

Also, we publish raw data and that make possible to compare it in longer period of time, this can be very usefully for statistical purposes and we can easy find big jumps in readings in a case for e.g. nuclear accidents.
____________
Regards,
Krzysztof 'krzyszp' Piszczek
Android Radioactive@Home Map
Android Radioactive@Home Map - donated
My Workplace

FogBRD
Send message
Joined: 16 Feb 12
Posts: 22
Credit: 6,008
RAC: 0
Message 780 - Posted: 27 Feb 2012 | 16:06:23 UTC - in response to Message 779.

My mother was a woman.

The mechanism of removal from the sensors (35 sec) changed inappropriately. I want to say that the loss of your base 45 times (instead of 1 GB to 22 Mb), the base will not lose relevance. In addition to store data that 60% are not authentic - why?
You only think about the figure - the information removed from the sensor by 60% INCORRECT.
As for the maps and graphics - to me a little that is clear. And the rest, too. Much clearer if the scale of the X will be the time. Hour mark, and half hour.
With regard to accidents involving radioactive material - it's about the collapse of HOURS and the level of contamination.
All I want to advise you - do your project is easy for people. Without loss of informativity and accuracy. But it will reduce your maintenance costs.
Well at least you read my posts. And I have a feeling that you do not want to listen to - hold opinions and to the point.

Tell me where am I wrong?
____________

Profile krzyszp
Project administrator
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 16 Apr 11
Posts: 368
Credit: 453,845
RAC: 141

Message 781 - Posted: 27 Feb 2012 | 17:29:53 UTC - in response to Message 780.

Extending measure time give you better accuracy - that's time but:

More data give more statistical information, in some cases that is important (e.g. in calculating standard deviation).
We spend long time discussing, how often sensor should report data and how accurate they are and on your point of view, extending sample time give more accurate data - that's right, but try to understand that project data can be usefully also for other users to calculate very different parameters?

I know, that we can save space, load and costs, but in my opinion on this way we cut some usefully information.


____________
Regards,
Krzysztof 'krzyszp' Piszczek
Android Radioactive@Home Map
Android Radioactive@Home Map - donated
My Workplace

FogBRD
Send message
Joined: 16 Feb 12
Posts: 22
Credit: 6,008
RAC: 0
Message 782 - Posted: 27 Feb 2012 | 18:32:23 UTC - in response to Message 781.


More data give more statistical information, in some cases that is important (e.g. in calculating standard deviation).

I agree with you if we were talking about counters SBT-10, si8bg who have sensitivity to 5 times higher. But the parameters are determined by a Geiger counter. And in this case it bad statistics (for example, think of the "own background = 1imp / s"). And detailed measurements of the base does not increase the accuracy and quality.
Anyway you socialize with people who are working with radioactive materials and measurements - at the mention of the GMS-20, they just laughed at you. Yes it is a popular counter, but it is reliable, simple, and the test of time. But do not expect more from him.
By the way if you really want to improve the accuracy and inexpensive to use the SBM-19 counter. price is the same.
____________

Alessandro Freda
Send message
Joined: 17 Aug 11
Posts: 36
Credit: 1,035,026
RAC: 448

Message 784 - Posted: 28 Feb 2012 | 13:22:15 UTC - in response to Message 776.
Last modified: 28 Feb 2012 | 14:00:28 UTC

Current database of results for only 60 detectors is bigger than 1GiB of data. This data are constantly updated and processed in system.

To save space and bandwidth on the server you could try to compress and distribute the raw data via some p2p network. Some company like ASUS use these networks as optional way to distribute their software manuals and drivers, to reduce the bandwidth cost of their servers.
So another contribution of some radioactive users ( that are also user of p2p networks like eMule or bit torrent ) could be hosting on their HDD (some or all) the historical raw data files of the project to provide many sources (and faster download) for the files.
You just be careful to publish the p2p links to these files on the site because I read that some ISP run scripts that automatically delete all sites that hold mp3 file or p2p links without any capability to check if the pointed files are copyrighted or not.

Profile ChertseyAl
Avatar
Send message
Joined: 16 Jun 11
Posts: 152
Credit: 385,578
RAC: 160

Message 785 - Posted: 28 Feb 2012 | 18:42:48 UTC - in response to Message 784.


To save space and bandwidth on the server you could try to compress and distribute the raw data via some p2p network


Sadly, a number of ISPs now utilise DPI and throttle p2p traffic making it very inefficient. Also, p2p has a bad name as so much contest *is* copyrighted material. I did once meet someone who downloaded 'legal' content via p2p. But only once :)

Al.

____________

Profile krzyszp
Project administrator
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 16 Apr 11
Posts: 368
Credit: 453,845
RAC: 141

Message 786 - Posted: 28 Feb 2012 | 18:52:04 UTC - in response to Message 785.


To save space and bandwidth on the server you could try to compress and distribute the raw data via some p2p network


Sadly, a number of ISPs now utilise DPI and throttle p2p traffic making it very inefficient. Also, p2p has a bad name as so much contest *is* copyrighted material. I did once meet someone who downloaded 'legal' content via p2p. But only once :)

Al.

I'm using torrents for downloading and uploading linux distros and also I have two TOR servers, I'm probably really bad person for authorities ;)

Anyway.
I will try option with p2p sharing archive data, probably hosted on one of my VPS's.
____________
Regards,
Krzysztof 'krzyszp' Piszczek
Android Radioactive@Home Map
Android Radioactive@Home Map - donated
My Workplace

FogBRD
Send message
Joined: 16 Feb 12
Posts: 22
Credit: 6,008
RAC: 0
Message 787 - Posted: 28 Feb 2012 | 19:05:15 UTC - in response to Message 786.

Torrent is good that the archival materials can be placed in your home that provides low-cost content. Many companies today distribute any software through its P2P paid and free. So it's a good idea.

P.S. Here, for example - http://www.mikrotik.com/download.html
____________

Dagorath
Avatar
Send message
Joined: 4 Jul 11
Posts: 151
Credit: 42,738
RAC: 0

Message 790 - Posted: 29 Feb 2012 | 0:21:00 UTC

Keeping 31 days of data in the database and providing the raw data in uncompressed format as described in this thread is unnecessary and it creates a huge demand for storage, database access time and bandwidth on the server. Why not clear the database daily and save the data in compressed files? People who want the data won't mind downloading compressed files. If you have a daily file for each continent (or region) it will allow users to choose only the data they want which would possibly decrease bandwidth.

I think FogBRD has raised some valid concerns and it sounds like he works in the nuclear industry or has some experience. You said you spent a lot of time discussing sample frequency but did you discuss that with radiation and contamination experts? I don't see any point in keeping more data than is necessary but you have said it allows people to calculate other parameters... what other useful parameters would people want to calculate?
____________

FogBRD
Send message
Joined: 16 Feb 12
Posts: 22
Credit: 6,008
RAC: 0
Message 792 - Posted: 29 Feb 2012 | 6:32:31 UTC - in response to Message 790.
Last modified: 29 Feb 2012 | 6:37:47 UTC

Indeed, what other data can be expected? I even became curious. Can you Show the world why you need to store redundant data does not have a scientific and practical value. That's just not talking about statistics. With regard to the SBM-20 is not justified. SBM-20 is not a proportional counter and can not provide the energy level of radiation as it makes a scintillation spectrometer. So we need time and level. Given its accuracy, and there are enough savings 5 minutes interval for the raw data and 30 minutes for the public on a map.
Here is an example of online dosimeter http://geiger.su/ click on the scale of 1h and look at the variations in time.

By the way Dagorath gave a good idea to do backups across regions.

I can the other - make a small wikipedia - we put things in order there soon (I hope users will be able to edit it).

The third idea - organize the formation of links to my point on the map.

And by the way is a link to the schedule of activities, many forums do not understand. It should be a link to the image rather than PHP that gives the picture. It would be a good advertisement for your signature on many forums and websites. As creators of a great public attention.

Speaking of birds - in many industrial dosimeters where there is a measurement of the background for further calculations, the length measurement of the background is from 5 to 30 minutes.
____________

Profile krzyszp
Project administrator
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 16 Apr 11
Posts: 368
Credit: 453,845
RAC: 141

Message 793 - Posted: 29 Feb 2012 | 7:33:44 UTC - in response to Message 792.

First point:
40s is basically sample time, but displayed value can be calculated for bigger time interval with more accuracy, but we can still save all samples.

Storing data for regions sounds interesting, but need some programming work to find a way how coordinates join with regions (I will try to find it out).



And by the way is a link to the schedule of activities, many forums do not understand. It should be a link to the image rather than PHP that gives the picture. It would be a good advertisement for your signature on many forums and websites. As creators of a great public attention.


Yes, we do this as well, just don't our free (for project) time is limited, this is way I have started discussion on this thread.
____________
Regards,
Krzysztof 'krzyszp' Piszczek
Android Radioactive@Home Map
Android Radioactive@Home Map - donated
My Workplace

FogBRD
Send message
Joined: 16 Feb 12
Posts: 22
Credit: 6,008
RAC: 0
Message 794 - Posted: 29 Feb 2012 | 8:42:17 UTC - in response to Message 793.

I want to bring that to counter this rather average values ​​every 5 minutes.

User is in my profile indicates his country - enough to make the archive by country.

____________

Profile krzyszp
Project administrator
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 16 Apr 11
Posts: 368
Credit: 453,845
RAC: 141

Message 796 - Posted: 29 Feb 2012 | 8:48:59 UTC - in response to Message 794.
Last modified: 29 Feb 2012 | 8:49:25 UTC

I'm indicated as Polish but live in UK and my detector is pointed in UK ;)
____________
Regards,
Krzysztof 'krzyszp' Piszczek
Android Radioactive@Home Map
Android Radioactive@Home Map - donated
My Workplace

FogBRD
Send message
Joined: 16 Feb 12
Posts: 22
Credit: 6,008
RAC: 0
Message 797 - Posted: 29 Feb 2012 | 9:53:29 UTC - in response to Message 796.

You seem to want to enter additional data in the user settings?
What prevents this is done.
- The location of the sensor - the country
- The location of the sensor - inside / outside the house
- Mode of operation - continuous / periodic
- Type of Geiger sbm-20/sbm-19 ...

and so on.
____________

Dagorath
Avatar
Send message
Joined: 4 Jul 11
Posts: 151
Credit: 42,738
RAC: 0

Message 800 - Posted: 1 Mar 2012 | 12:38:18 UTC - in response to Message 793.

Storing data for regions sounds interesting, but need some programming work to find a way how coordinates join with regions (I will try to find it out).


I will speak with some cartographers to see if there is an automated way to do this. Somebody at Google maps would know. If not then the manual method is easy too. You don't have a lot of new detectors joining the project every day so all it needs is a program to check if a detector is associated with a country/region, if it is then check if the coordinates have changed. If the coordinates have changed by more than x minutes or the detector has not been assigned to a region then a human determines the country/region manually and enters the country/region. Then every day the program adds that detector's data to the appropriate archive. With such a method each device can even be associated with a city, town, county, province or state.

I could program that and locate 20 detectors on the map per hour. If you ship 100 new detectors they could be handled manually in just 5 hours. So I'm just going to do it and maintain it. I'll show you when it's done. I'll get the data by scraping the HTML from the link in that other thread unless you have a better way.

In the meantime please consider getting some advice from someone in the nuclear industry regarding sample rate and possibly changing to a different GM tube.

____________

FogBRD
Send message
Joined: 16 Feb 12
Posts: 22
Credit: 6,008
RAC: 0
Message 801 - Posted: 1 Mar 2012 | 13:22:21 UTC - in response to Message 800.
Last modified: 1 Mar 2012 | 13:29:26 UTC

Switching to another type of Geiger counter is simple. The closest is an inexpensive SBM-19. Having thus a better performance is comparable to the price of SBM-20. All other counters are much more expensive.
Mica counters to use this excess. Of course they will see and alpha radionuclides in the drop rain but it greatly increases the cost of the sensor.

for information - a modern analogue of the SBM-19 is the STS-6
____________

Dagorath
Avatar
Send message
Joined: 4 Jul 11
Posts: 151
Credit: 42,738
RAC: 0

Message 803 - Posted: 1 Mar 2012 | 22:19:53 UTC - in response to Message 801.

Mica counters to use this excess. Of course they will see and alpha radionuclides in the drop rain but it greatly increases the cost of the sensor.


That is exactly what I want... a detector with mica window on tube to detect alpha radionuclides so I can check the rain, the soil and my food. I don't trust my government to report that information correctly.

Can I buy a tube with mica window and will it work with this project's version 1 detector (the version in the first shipment)? If necessary I can hack the detector and/or reprogram it to make it work.

____________

Profile krzyszp
Project administrator
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 16 Apr 11
Posts: 368
Credit: 453,845
RAC: 141

Message 804 - Posted: 1 Mar 2012 | 22:49:53 UTC - in response to Message 803.

Basically, you can use "any" tube with our detector, just new tube have to work with 400V power and obviously, results sent to server (if you change to another tube) will be useless in Rad@H project (but you can mark temporary your sensor as 'experimental' in your profile and feel free to find other targets for your sensor).

Answering for recent posts. I will try to manage regions as soon as we finish shipment of current batch, just we are to busy now.

One of the best point of Open Software (data, knowledge, etc) is that even if you don't predict all reasons why and for what people like to use your work, they can find something interesting. Recently one Polish student ask me for permission to download list of all impulse with times (without any other information from database) because he want to use it as part of his science work about random numbers generators... I know, this is not join on any way with radioactivity but shows power of open knowledge ;)

Also apologise for all delays in my answers, sometimes I have to do my paid work ;)
____________
Regards,
Krzysztof 'krzyszp' Piszczek
Android Radioactive@Home Map
Android Radioactive@Home Map - donated
My Workplace

Dagorath
Avatar
Send message
Joined: 4 Jul 11
Posts: 151
Credit: 42,738
RAC: 0

Message 805 - Posted: 2 Mar 2012 | 0:29:13 UTC - in response to Message 804.

I have 2 of your detectors so one would use the standard tube and stay connected to this project, the other would use the special tube and not be connected or be designated experimental.

Yes, there are many uses for the data and the decision what to do is yours. You will receive my support no matter what you decide. I was only making suggestions on how to save some money because the project does not have a lot of money. I won't pursue that topic any further.

Don't worry about delayed responses, your work and family come first. I am just happy you can find a little time to discuss and respond because so many project admins do not. Maybe they are too busy, no problem.

I am working on sorting and storing the data by region (country and possibly city or province) too.

____________

FogBRD
Send message
Joined: 16 Feb 12
Posts: 22
Credit: 6,008
RAC: 0
Message 808 - Posted: 2 Mar 2012 | 7:02:34 UTC - in response to Message 805.
Last modified: 2 Mar 2012 | 7:10:19 UTC

If you can compile from source code and program the controller is sufficient to change the formula calculation. Go tell that to measure the contamination of food is necessary to increase the time up to 5 - 30 min, after measuring the background for 30 minutes.


The formula for calculating the level that I apply:

rad_v = (rad_count * (3600/tim)) / sen

where:
rad_v - The result of the level of radioactivity
rad_count - The number of pulses during the measurement
tim - the time of measurement
seni - counter sensitivity

In the scheme I have seen loose pins 2,3,11 which can be used as controls. Using an external battery to 5V so you get a portable meter for measuring free anywhere.
Buttons should be used to zoom in and out of time measurement, configuration, and metering the background.
____________

Post to thread

Message boards : News : Server change requirments


Main page · Your account · Message boards


Copyright © 2019 BOINC@Poland | Open Science for the future