Read a Delimited File
How can you read a row in delimited file and get each field value into a separate variable?
#!/usr/bin/bash IFS=":" while read f1 f2 f3; do echo "$f1-$f2-$f3" done < delim.txt
Using the following input file as a sample:
field1:field2:field3 field1a:field2a:field3a
Sample run:
$ ./delim.sh field1-field2-field3 field1a-field2a-field3a
PHP mail and ISP relays
This did my head in so I thought it worthy of a small write up.
Here the problem: I wanted to send an email from a DRUPAL 5.12 site hosted on a home linux configured with sendmail/postfix and a SMART RELAYHOST.
Sounds simply enough.
Here is my simple test program in PHP to send an email.
<?php $Name = "Da Duder"; $email = "dude@duder.com"; $recipient = "yo@company.com"; $mail_body = "The text for the mail..."; $subject = "Subject for reviewer"; $header = "From: ". $Name . " <" . $email . ">\r\n"; mail($recipient, $subject, $mail_body, $header); ?>
Doesn't get easier than that. Now this is where the fun starts. If you look at the mail log.
/var/log/maillog
Dec 3 16:05:53 elmo postfix/pickup[13979]: 06BCA92EE80: uid=48 from=<apache> Dec 3 16:05:53 elmo postfix/cleanup[14081]: 06BCA92EE80: message-id=<20081203160553.06BCA92EE80@elmo.local> Dec 3 16:05:53 elmo postfix/qmgr[13980]: 06BCA92EE80: from=<apache@elmo.local>, size=330, nrcpt=1 (queue active) Dec 3 16:05:53 elmo postfix/smtp[14083]: 06BCA92EE80: to=<yo@company.com>, relay=smtp.blueyonder.co.uk[195.188.53.60]:25, delay=0.21, delays=0.06/0.02/0.07/0.06, dsn=2.0.0, status=sent (250 OK id=1L7sEh-0000Yg-Kl) Dec 3 16:05:53 elmo postfix/qmgr[13980]: 06BCA92EE80: removed
You see that the envelope-sender is actually apache@elmo.local this address is checked by yo@company.com spam's filter says “Hay that not valid I'll reject that”.
So what you need to do is tell sendmail that it should use the “from:” as the envelope sender too.
Small modification to the PHP and this is done. Why did that take me an hours to figure out? God only knows.
mail($recipient, $subject, $mail_body, $header,'-f'.$email);
So back to DRUPAL the PHP mail() function is wrapped by the function drupal_mail() in the includes/common.inc file.
So we tweak the invokation like so:
function extract_email_from($string){ preg_match_all("/[\._a-zA-Z0-9-]+@[\._a-zA-Z0-9-]+/i", $string, $matches); return $matches[0][0]; } function drupal_mail($mailkey, $to, $subject, $body, $from = NULL, $headers = array()) { .. snip .. return mail( $to, mime_header_encode($subject), str_replace("\r", '', $body), join("\n", $mimeheaders), '-f '.extract_email_from($defaults['From']) ); }
Reference: http://drupal.org/node/131737, http://codesnippets.joyent.com/posts/show/179
JIT Data Fetching
One of the problem with using a datagrid is that if you simply bind a “select * from emp” dataset to it then all rows from the EMP table will be retrieved and rendered – this is just not workable for tables with a large number of rows.
This is the Microsoft implementation of this the virtual just in time fetcher for SQL*Server from which I based my Oracle implementation on.
Implementing Virtual Mode with Just-In-Time Data Loading in the Windows Forms Data Grid View Control. http://msdn2.microsoft.com/en-us/library/ms171624
Changes I had to make to get this working with an Oracle database:
The SQL*Server connection string in the Form1.cs file needs to be changed to an Oracle connection string.
connectionString = "Data Source=muon;Persist Security Info=True;User ID=scott;"+ "Password=tiger;Unicode=False;Omit Oracle Connection Name=True";
I created a database table emp_large in the scott/tiger user that contains 14,000+ rows of data for testing purposes:
private string table = "emp_large";
All references to the following classes must be changed:
SqlCommand -> OracleCommand SqlConnection -> OracleConnection SqlDataAdapter -> OracleDataAdapter SqlException -> OracleException
and where ever these changes occur the following reference must also be modified
using System.Data.SqlClient; -> using System.Data.OracleClient;
If we were trying to be database agnostic then these classes should all be replaced with Factory methods and some sort of database abstraction layer.
The following changes all appears in the C# file “DataRetreiver.cs” this module is response for physically fetching data from the Oracle database.
public int RowCount { get { // Return the existing value if it has already been determined. if (rowCountValue != -1) { return rowCountValue; } // Retrieve the row count from the database. command.CommandText = "SELECT COUNT(*) FROM " + tableName; rowCountValue = (int)command.ExecuteScalar(); return rowCountValue; } }
This implementation should also work with the Oracle database but it was found that is does not with out the following change
rowCountValue = Convert.ToInt16(command.ExecuteScalar());
The reason for this is that I suspect the Execute Scalar method returns an object but when used with the Oracle drivers C# can't autobox the conversion and must be told explicity. Very ODD.
The most substantial modification is the sql statement that fetches a limited number of rows from the database, the SQL Server code appears below:
command.CommandText = "Select Top " + rowsPerPage + " " + CommaSeparatedListOfColumnNames + " From " + tableName + " WHERE " + columnToSortBy + " NOT IN (SELECT TOP " + lowerPageBoundary + " " + columnToSortBy + " From " + tableName + " Order By " + columnToSortBy + ") Order By " + columnToSortBy; adapter.SelectCommand = command; DataTable table = new DataTable(); table.Locale = System.Globalization.CultureInfo.InvariantCulture; adapter.Fill(table);
and the Oracle implementation. To improve SGA caching I have introduced bind variables where the paging variables are needed. As the command (Oracle Command) object is reused in other parts of the code after the adapter.Fill() method has executed the parameters must be cleared to prevent them being applied to other statements. Additional exception handling was included to aid debugging.
// Retrieve the specified number of rows from the database, starting // with the row specified by the lowerPageBoundary parameter. try { command.CommandText = "select * "+ "from (select p.*,rownum rnum "+ "from (select * from "+tableName+ " order by "+columnToSortBy+") P "+ "where rownum <= :pROWNUM ) " + "where rnum > :pRNUM " + "order by "+columnToSortBy; command.Parameters.Add("pROWNUM", OracleType.Number).Value = rowsPerPage+lowerPageBoundary; command.Parameters.Add("pRNUM", OracleType.Number).Value = lowerPageBoundary; adapter.SelectCommand = command; DataTable table = new DataTable(); table.Locale = System.Globalization.CultureInfo.InvariantCulture; adapter.Fill(table); command.Parameters.Clear(); ///<--- As the command object is reused elsewhere } catch (OracleException oe) { System.Windows.Forms.MessageBox.Show("Oracle Exception: "+oe.Message+ "\n\n"+command.CommandText); throw; }
The idea for the SQL query came from a Web solution of the problem.
A sorting and paging Data Grid loading data on-demand, saving changes in bulk from and to an Oracle database, in ASP.NET. http://www.codeproject.com/aspnet/OracleDataGrid.asp
HP MediaSmart Home Server EX475
With only 512Mb of memory the write performance does suffer. After doing a bit of reading on this forum thread.
I decided to upgrade the memory
- Crucial part CT25664AA667
- or from Ebuyer - http://www.ebuyer.com/product/142383
This made a phenomenal difference before I was only getting a max of 20Mb/s on writes now I get 35Mb/s due to all the additional cache. W00t!
Orange LiveBox Terminal Server?
Would it be interesting to be able to hack the Orange liveBox and install some software that would allow the serial port to be accessed remotely. This would make a good single port terminal server.
If the USB ports can drive external HDD's then it may also be a cheap NAS.
- Telnet on the LiveBox - http://bluetoad.net/livebox/
- Serial over LAN daemon - http://www.acmesystems.it/?id=69
- French site - http://www.porciello.com - English translate
This has been promoted to its own wiki page now livebox hacking