Exit
  • Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
0

How to get data into the mySQL database?

LEGEND ,
May 19, 2011 May 19, 2011

Copy link to clipboard

Copied

First some background.

I have a website that has outgrown its designed dimensions and is a huge burden to maintain. See PPBM5 Benchmark

There is a lot of maintenance work involved, so I'm investigating a PHP/MySQL approach to easen the burden and to add functionality to the site. With the current Excel based structure and over 420 entries, it is cumbersome for me to maintain, but also for users to find what they need.

A MySQL based dynamic structure is a lot easier and offers vastly more selection capabilities, like selecting only records that meet specific criteria.

Data submission is done with a form, that contains most of the relevant data, but the drawack is that people submitting their data are often not technically inclined, give wrong answers due to a lack of understanding or making typo's. The test results are attached in one or two separate .txt files, but often they have not read the instructions correctly or did something wrong, so these attached .txt files can not be trusted automatically, they have to be checked before inclusion.

These were my initial thoughts:

1. Data collection:

To avoid spending all our energy and time  on correcting typo's, getting missing data, correcting errors, I am  investigating the use of CPU-Z in Ghost mode to create a .txt or .html  file that contains all relevant hardware info we need and even more. It gives all the info we currently have, but adds  data like number of memory sticks, DDR timings, stock clock speed and  BCLK setting, video card info and VRAM size, etc.

To see what I mean, run CPU-Z, go to the About tab and press the Save Report button and look at the results.

This can all be done without user intervention in an automatic way, but  maybe I need to add an Auto-It file to the test to make it all run as  desired.

If this works and I'm able to extract the relevant data from the created  file and can insert it into the database, we may be in business for the  next version of PPBM5.5 or PPBM6. It does require a modification to the instructions, making them a lot  easier, because there is less data to fill out.

2. Data submission:

The submission form can be simplified if  the CPU-Z data can be used. We have to create an automatic way to attach  the created .html file from CPU-Z to the submission form and we have to  streamline the Output.txt and Output-MPE.txt files to be more easily included in the 'form.lib.php' file. It  currently is manual labor and very time consuming.

3. Adding to Database:

I have to find a way to create database  records from the Gmail forms I receive. All incoming mail messages need  to be checked on relevancy and if relevant, need to be added  automatically to the database and then offered for approval before final inclusion in the database. Data included in the database  will then include submission date and time, Email address,  IP address  used, plus links to the files submitted and available on the website.

4. Publication of the database:

After approval of new records from step  3, all updates will be automatically applied to the database and  accessible for users. I do not yet intend to introduce a user account ,  requesting login before all functionality is accessible. Too much trouble and administration.

Queries should be possible on things like CPU (check box), so include  17-920, i7-930, i7-950 but exclude i7-980X and i7-990X, Size of memory  (check box), Overclocked (boolean, yes, no), SSD as OS disk, and similar  options.

The biggest problem is to keep the color grading and statistical  indicators (Top, D9, Q3, Med, Q1 and D1) intact on dynamically generated  queries. Say you make a query which results in 20 observations, this  should show the related colors and legends. Next query results in 48 observations and of course the color grading and legends  do need to reflect that. Question in my mind, does the RPI remain  constant, independent of the query or does that need to be recalculated  on the basis of the query?

Next thing is to allow a user to select a specific observation and by  simply clicking on it be shown, in a separate window (detail page) or  accordion, all the CPU-Z related information about the hardware.

The graphs, Top-20 and MPE Gains, need to be dynamically adjusted, based on the query used.

5. Ideally, external links:

In an ideal situation, one could link the  CPU-Z data to external price databases, looking up current prices for  CPU, memory, video card, disks, raid controller, etc. to get instant  BFTB charts, based on the query made. But that is the next step.

Situation now:

I have a MySQL database that is easily updated with the new submissions. Simply create a .CSV flie from the submitted forms and import that into the database. The bulk of the initial work is done.Lots remain to be done as you can see above, but that is for a later time.

Question:

I have this table, that needs to be filled with data in the submitted and attached files. Mr. X submitted his data and can be uniquely identified by his "Ref_ID". He attached one or two files in .TXT format with the relevant test data. These files are stored on the server with a concatenated name:

"Ref_ID","-","filename"

Say his Ref-ID is: 20110204-6cf5 and his submitted file is called: Output(99).txt then the file can be found on the server as

20110204-6cf5-Output(99).txt

19-5-2011 12-45-44.png
I need to be able to open that comma delimited file, the contents may look like this: "439","1036","819","531" and insert these contents into the relevant record and fields.

Graphically,

19-5-2011 12-44-02.png

is what I want to achieve.

This being my first exposure to PHP/MySQL, you can imagine I'm not clear on how to go from here.

Added complication is that I actually have 5 numbers to insert per record and two calculated fields, Total Score and RPI should be calculated fields. Haven't yet figured out how to handle calculated fields, maybe only in the PHP/HTML code and not in the database.

I hope someone can help me.


TOPICS
Server side applications

Views

1.1K
Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
May 26, 2011 May 26, 2011

Copy link to clipboard

Copied

No suggestions at all???

Votes

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
May 26, 2011 May 26, 2011

Copy link to clipboard

Copied

Harm,

I have read your problem before and I am glad  to hear that you are moving your data in this direction.  However, due  to the size and complexity of your situation and the necessary time that  would need to be devoted to such a project, I believe, and believe that  others may feel the same, that your project warrants outsourced/contract help.  The amount of data you maintain, plus the migration and then the programming of a backend to replace Excel requires full-time attention (albiet temporary).  Not many topics on these forums that go that in depth last long because it scares the average poster away when many do it for a living and have other large projects to attend to.

Votes

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
May 30, 2011 May 30, 2011

Copy link to clipboard

Copied

You do have a very complex looking site and may need several tables in mysql to handle all that data. If you knew to phpmysql I would suggest taking a look at this tutorial it will help get you started in understanding how to $_GET info from a database and also how to $_POST data to a database. I am no expert just learning myself and I found this very helpful. This is the link http://www.adobe.com/devnet/dreamweaver/articles/first_dynamic_site_pt1.html

There are also many tutorials on Youtube to help build a CMS Content Management Site I would suggest the following: -

http://www.youtube.com/user/phpacademy

http://www.youtube.com/user/betterphp

http://www.youtube.com/user/flashbuilding

And many more on my channel here

http://www.youtube.com/user/Whisperingonthewind

CMS's are easier to maintain, add edit and delete content.

I have also recently bought a Book by David Powers Training from the Source very helpful.

Anyway hope you get it sorted.

Votes

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Jun 07, 2011 Jun 07, 2011

Copy link to clipboard

Copied

LATEST

Update: With a lot of manual labour I got all the existing data into the MySQL database and for new submissions it is now handled automatically.

Data are now automatically added to the database like this:

8-6-2011 2-27-39.png

and the individual results are reported back to the submitter.

Two major hurdles to take now:

1. Importing 'old' - non-form based - submissions into the database, and

2. Designing the pages to access the database.

After that, it is all cosmetics. Adding the statistical measures and color gradients, based on the query, etc.

Votes

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines