tidbits

Read a Delimited File

How can you read a row in delimited file and get each field value into a separate variable?

#!/usr/bin/bash
 
IFS=":"
while read f1 f2 f3; do
  echo "$f1-$f2-$f3"
done < delim.txt

Using the following input file as a sample:

field1:field2:field3
field1a:field2a:field3a

Sample run:

$ ./delim.sh
field1-field2-field3
field1a-field2a-field3a
2009/11/27 17:53

PHP mail and ISP relays

This did my head in so I thought it worthy of a small write up.

Here the problem: I wanted to send an email from a DRUPAL 5.12 site hosted on a home linux configured with sendmail/postfix and a SMART RELAYHOST.

Sounds simply enough.

Here is my simple test program in PHP to send an email.

<?php
$Name = "Da Duder";
$email = "dude@duder.com";
$recipient = "yo@company.com";
$mail_body = "The text for the mail...";
$subject = "Subject for reviewer";
$header = "From: ". $Name . " <" . $email . ">\r\n";
 
mail($recipient, $subject, $mail_body, $header);
?>

Doesn't get easier than that. Now this is where the fun starts. If you look at the mail log.

Mail log says postfix. I thought the problem might have been to do with my sendmail configuration so I installed postfix. It turns out that it wasn't related both suffer from the same fate.

/var/log/maillog

Dec  3 16:05:53 elmo postfix/pickup[13979]: 06BCA92EE80: uid=48 from=<apache>
Dec  3 16:05:53 elmo postfix/cleanup[14081]: 06BCA92EE80: message-id=<20081203160553.06BCA92EE80@elmo.local>
Dec  3 16:05:53 elmo postfix/qmgr[13980]: 06BCA92EE80: from=<apache@elmo.local>, size=330, nrcpt=1 (queue active)
Dec  3 16:05:53 elmo postfix/smtp[14083]: 06BCA92EE80: to=<yo@company.com>, relay=smtp.blueyonder.co.uk[195.188.53.60]:25, delay=0.21, delays=0.06/0.02/0.07/0.06, dsn=2.0.0, status=sent (250 OK id=1L7sEh-0000Yg-Kl)
Dec  3 16:05:53 elmo postfix/qmgr[13980]: 06BCA92EE80: removed

You see that the envelope-sender is actually apache@elmo.local this address is checked by yo@company.com spam's filter says “Hay that not valid I'll reject that”.

So what you need to do is tell sendmail that it should use the “from:” as the envelope sender too.

Small modification to the PHP and this is done. Why did that take me an hours to figure out? God only knows.

mail($recipient, $subject, $mail_body, $header,'-f'.$email);

So back to DRUPAL the PHP mail() function is wrapped by the function drupal_mail() in the includes/common.inc file.

So we tweak the invokation like so:

function extract_email_from($string){
  preg_match_all("/[\._a-zA-Z0-9-]+@[\._a-zA-Z0-9-]+/i", $string, $matches);
  return $matches[0][0];
}
 
function drupal_mail($mailkey, $to, $subject, $body, $from = NULL, $headers = array()) {
 
.. snip ..
 
    return mail(
      $to,
      mime_header_encode($subject),
      str_replace("\r", '', $body),
      join("\n", $mimeheaders),
      '-f '.extract_email_from($defaults['From'])
    );
}

Reference: http://drupal.org/node/131737, http://codesnippets.joyent.com/posts/show/179

2009/11/27 17:53

JIT Data Fetching

One of the problem with using a datagrid is that if you simply bind a “select * from emp” dataset to it then all rows from the EMP table will be retrieved and rendered – this is just not workable for tables with a large number of rows.

This is the Microsoft implementation of this the virtual just in time fetcher for SQL*Server from which I based my Oracle implementation on.

Implementing Virtual Mode with Just-In-Time Data Loading in the Windows Forms Data Grid View Control. http://msdn2.microsoft.com/en-us/library/ms171624

Changes I had to make to get this working with an Oracle database:

The SQL*Server connection string in the Form1.cs file needs to be changed to an Oracle connection string.

connectionString = "Data Source=muon;Persist Security Info=True;User ID=scott;"+
                     "Password=tiger;Unicode=False;Omit Oracle Connection Name=True";

I created a database table emp_large in the scott/tiger user that contains 14,000+ rows of data for testing purposes:

private string table = "emp_large";

All references to the following classes must be changed:

        SqlCommand -> OracleCommand
        SqlConnection -> OracleConnection
        SqlDataAdapter -> OracleDataAdapter
        SqlException -> OracleException

and where ever these changes occur the following reference must also be modified

  using System.Data.SqlClient; -> using System.Data.OracleClient;

If we were trying to be database agnostic then these classes should all be replaced with Factory methods and some sort of database abstraction layer.

The following changes all appears in the C# file “DataRetreiver.cs” this module is response for physically fetching data from the Oracle database.

  public int RowCount
    {
        get
        {
            // Return the existing value if it has already been determined.
            if (rowCountValue != -1)
            {
                return rowCountValue;
            }
 
            // Retrieve the row count from the database.
            command.CommandText = "SELECT COUNT(*) FROM " + tableName;
            rowCountValue = (int)command.ExecuteScalar();
            return rowCountValue;
        }
    }

This implementation should also work with the Oracle database but it was found that is does not with out the following change

rowCountValue = Convert.ToInt16(command.ExecuteScalar());

The reason for this is that I suspect the Execute Scalar method returns an object but when used with the Oracle drivers C# can't autobox the conversion and must be told explicity. Very ODD.

The most substantial modification is the sql statement that fetches a limited number of rows from the database, the SQL Server code appears below:

 command.CommandText = "Select Top " + rowsPerPage + " " +
 CommaSeparatedListOfColumnNames + " From " + tableName +
            " WHERE " + columnToSortBy + " NOT IN (SELECT TOP " +
            lowerPageBoundary + " " + columnToSortBy + " From " +
            tableName + " Order By " + columnToSortBy +
            ") Order By " + columnToSortBy;
 adapter.SelectCommand = command;
 
 
 DataTable table = new DataTable();
 table.Locale = System.Globalization.CultureInfo.InvariantCulture;
 adapter.Fill(table);

and the Oracle implementation. To improve SGA caching I have introduced bind variables where the paging variables are needed. As the command (Oracle Command) object is reused in other parts of the code after the adapter.Fill() method has executed the parameters must be cleared to prevent them being applied to other statements. Additional exception handling was included to aid debugging.

  // Retrieve the specified number of rows from the database, starting
  // with the row specified by the lowerPageBoundary parameter.
  try
  {
    command.CommandText = "select * "+
                        "from (select p.*,rownum rnum "+
                              "from (select * from "+tableName+
                                     " order by "+columnToSortBy+") P "+
                              "where rownum <= :pROWNUM ) " +
                         "where rnum > :pRNUM " +
                         "order by "+columnToSortBy;
 
 
    command.Parameters.Add("pROWNUM", OracleType.Number).Value = rowsPerPage+lowerPageBoundary;
    command.Parameters.Add("pRNUM", OracleType.Number).Value = lowerPageBoundary;
 
 
    adapter.SelectCommand = command;
 
 
    DataTable table = new DataTable();
    table.Locale = System.Globalization.CultureInfo.InvariantCulture;
    adapter.Fill(table);
    command.Parameters.Clear();       ///<--- As the command object is reused elsewhere
  }
  catch (OracleException oe)
  {
      System.Windows.Forms.MessageBox.Show("Oracle Exception: "+oe.Message+
                                           "\n\n"+command.CommandText);
      throw;
  }

The idea for the SQL query came from a Web solution of the problem.

A sorting and paging Data Grid loading data on-demand, saving changes in bulk from and to an Oracle database, in ASP.NET. http://www.codeproject.com/aspnet/OracleDataGrid.asp

2009/11/27 17:53

HP MediaSmart Home Server EX475

CREATOR: gd-jpeg v1.0 (using IJG JPEG v62), default quality With only 512Mb of memory the write performance does suffer. After doing a bit of reading on this forum thread.

I decided to upgrade the memory

This made a phenomenal difference before I was only getting a max of 20Mb/s on writes now I get 35Mb/s due to all the additional cache. W00t!

2009/11/27 17:53

Orange LiveBox Terminal Server?

Would it be interesting to be able to hack the Orange liveBox and install some software that would allow the serial port to be accessed remotely. This would make a good single port terminal server.

If the USB ports can drive external HDD's then it may also be a cheap NAS.

This has been promoted to its own wiki page now livebox hacking

2009/11/27 17:53

<< Newer entries | Older entries >>

  • tidbits.txt
  • Last modified: 2009/11/27 16:59
  • by 127.0.0.1