Key Cloud Service Architecture Considerations

Converting an existing product or service for use in the cloud should begin with a consideration of several key requirements. Simply extending a system to support more clients may result in serious security, performance, and scalability issues.

These core requirements should be addressed prior to the application specific features, they represent the foundation for the system. These common elements apply for virtually any cloud based service.

  • Secure - The system must be designed and developed with best practices for security.
  • Scalable - The system must be scalable, to allow additional organizations to join without impacting others. The system capacity must be able to grow as necessary.
  • Manageable - It must be possible to adjust the resource allocation in response to changing consumption. If one server is too busy, clients should be able to be moved to a different server, with minimal service impact.
  • Fault Tolerant - System outages should be minimized and a disaster recovery plan must be in place. This includes backup and restore procedures, whether the failure or problem is due to a hardware, software, administrator, or user issue.
  • Segmented - Each client or organization must have a separate file system area, or account, and database, and access to these resources should limited as necessary for the use of the data. Failure to organize and protect the data can make it difficult or impossible to secure, scale and manage. Although some service elements may be system or server based, they must be provisioned or partitioned for use in such a way that the data is segmented.
  • Access Paths - Cloud services should allow subscribers to white label, so they can offer the services as if they are their own. Access URLs should be defined by the subscribers, probably as subdomains or derivative domains from their existing architecture.
  • Brandable - It is important that the services be brandable, so clients can identify the system as theirs. The extent of branding can vary, but at a minimum, a corporate logo should be displayed, and some connections or links to the corporate site and support resources should be provided. A neutral color scheme, such as grayscale can make it easier to ensure a nice presentation regardless of the logo.
  • Incremental - A tiered service offering should be established to allow free trials, as well as service levels so clients can purchase only the services they need.
  • Extensible - Cloud offerings should include the ability to extend and integrate with other systems, and those extensions should be managed at the client level. Thus, if one subscriber needs custom code, the code can be added without affecting other clients.
  • Integration - As with extensions, custom integration should be enabled to allow access into client infrastructures without affecting other clients.
  • Turnkey - The base system should be available very quickly, set up time should be minimal.
  • Easy to Use - Examine any existing systems and consider support requests to identify areas that have been difficult for clients to work with. Be sure to consider ways to improve these issues before proceeding.
  • Web Service or API Access - Subscribers should be able to interact with the system through automated processes with standards based interfaces such as SOAP and REST. These interfaces must be well-defined and documented. Test code and interfaces should be provided to ease development.
  • Well Supported - An extension of easy to use, support refers to ensuring users can use the system to do what needs to be done in a timely manner. The application needs to be organized carefully, supported with validation and help on the client (browser) side, adequate documentation, meaning a full manual in PDF, HTML or .zipped, a ticket system and possible a live chat interface.
  • Cost Effective - The cost of the system must be competitive.
  • Service Level Agreement - A service level agreement must be provided. This ensures both the provider and client understand exactly what is being offered.
  • Localization (l10n)
  • Internationalization (i18n)

Word 2007 VBA Macro to Create PDFs for a Manual

This is a macro that reads a (Microsoft) Word 2007 document which uses INCLUDETEXT fields to draw in content from other documents and exports the content as a PDF, including a table of contents and index.

There are three administrator types or roles - System, Partner, and Limited. For each chapter, an INCLUDETEXT field includes the content for that role, for example:

{INCLUDETEXT "{BaseDir}//Overview//{AdminRole}.docx"}

The document for each role has the appropriate content. In most cases, the directory has a Common.docx file that has the text for all roles, and those authorized to view it include the file like so:

{INCLUDETEXT "{BaseDir}//Overview//Common.docx"}

Unauthorized users would have an empty .docx file, or one with a limited version of content.

The way the macro works is to first extract the value of BaseDir, which allows an absolute path to be used, but modified without updating all the files. Relative paths just didn’t work well.

The document also uses some bookmarks, which the macro uses to reference different sections. One bookmark includes all the content from both the chapters and appendices. The chapters are enclosed in a bookmark called ‘Chapter’, and the appendices are enclosed in a bookmark called ‘Appendix’. These bookmarks are used to create the footers, since the chapters are numeric and the appendices are alphabetic. If content for the chapter is included for the role, a section break is inserted. This ensures that the chapter and page numbers are consequetive, even if chapters are omitted.



Option Base 1
Option Explicit
Global AdminTypes

Sub Init()
    AdminTypes = Array("System", "Partner", "Limited")
End Sub

Sub CreatePDFs()

Dim C, D, I, L, N, T, F, BaseDir, FieldCount
Dim BaseDirFieldItemIndex, AdminTypeFieldItemIndex, CurrentYearFieldItemIndex
Dim rngTemp As Range, rngField As Range
Dim fldPtr As Field
Dim J, JL, JU, arrBookmarks(2) As String
Dim Selection As Range
Dim Footer As Range
Dim S As String, LastFooterType As String

Init
I = 1
AdminTypeFieldItemIndex = 0
BaseDirFieldItemIndex = 0

FieldCount = ActiveDocument.Fields.Count()

For I = 1 To FieldCount
    T = ActiveDocument.Fields.Item(I).Code.Text
    If (InStr(1, T, "BaseDir", vbTextCompare) <> 0) Then
        BaseDirFieldItemIndex = I
    End If
    If (InStr(1, T, "AdminType", vbTextCompare) <> 0) Then
        AdminTypeFieldItemIndex = I
    End If
    If (InStr(1, T, "CurrentYear", vbTextCompare) <> 0) Then
        CurrentYearFieldItemIndex = I
    End If
    If AdminTypeFieldItemIndex <> 0 And BaseDirFieldItemIndex <> 0 And CurrentYearFieldItemIndex <> 0 Then Exit For
Next

' Set the current year for the copyright date
Set rngTemp = ActiveDocument.Fields.Item(CurrentYearFieldItemIndex).Code
rngTemp.Text = " SET CurrentYear " + Chr(34) + Str$(DatePart("yyyy", Date)) + Chr(34) + " "

' Set the base directory for INCLUDETEXT tags
ActiveDocument.Fields.Item(BaseDirFieldItemIndex).Update
D = ActiveDocument.Fields.Item(BaseDirFieldItemIndex).Result()
Set rngTemp = ActiveDocument.Fields.Item(BaseDirFieldItemIndex).Code
rngTemp.Text = " SET BaseDir " + Chr(34) + D + Chr(34) + " "
D = D + Chr(92)

' Setup the loop boundaries for the roles
I = LBound(AdminTypes)
L = UBound(AdminTypes)

' Set up the sections that will be processed
arrBookmarks(1) = "Chapter"
arrBookmarks(2) = "Appendix"
JL = LBound(arrBookmarks)
JU = UBound(arrBookmarks)
LastFooterType = ""

' Loop through all the admin types or roles
For N = I To L

    ' Set the role for this document
    Set rngTemp = ActiveDocument.Fields.Item(AdminTypeFieldItemIndex).Code
    rngTemp.Text = " SET AdminType """ + AdminTypes(N) + """ "
    ActiveDocument.Fields.Item(AdminTypeFieldItemIndex).Update
         
    ' Update all the fields for this role
    Set rngTemp = ActiveDocument.Bookmarks("ChapterBlockStart").Range
    rngTemp.Select
    rngTemp.Fields.Update
    
    ' Loop through the sections that will be processed.
    ' Each included file is checked to see if content was included
    ' Files that have content are followed by a section break, empty files are not
    For J = JL To JU
         
        Set rngTemp = ActiveDocument.Bookmarks(arrBookmarks(J)).Range
        rngTemp.Select
     
        For Each fldPtr In rngTemp.Fields
            ' Loop through all the field in this section or group of included files
            T = fldPtr.Type
            ' If this field is an INCLUDETEXT
            If (T = wdFieldIncludeText) Then
                T = Trim(fldPtr.Result())
                ' If the included text is not empty
                If (T <> "") And (Asc(T) <> 13) Then
                    Set rngField = ActiveDocument.Range
                    rngField.Find.Text = fldPtr.Code
                    ' Search the document for the tag.  This ensures included tags do not add section breaks
                    rngField.Find.Execute
                    If rngField.Find.Found Then
                        ' Select the tag
                        rngField.Select
                        ' Advance the range to the end of the field
                        rngField.MoveEnd wdCharacter, 2
                        rngField.Collapse wdCollapseEnd
                        ' Page numbering starts at 1 for all sections
                        rngField.Sections(1).Footers(wdHeaderFooterPrimary).PageNumbers.StartingNumber = 1
                        rngField.Sections(1).Footers(wdHeaderFooterPrimary).PageNumbers.RestartNumberingAtSection = True
                        If LastFooterType = arrBookmarks(J) Then
                            rngField.Sections(1).Footers(wdHeaderFooterPrimary).LinkToPrevious = True
                        Else
                           rngField.Sections(1).Footers(wdHeaderFooterPrimary).LinkToPrevious = False
                            ' Create new footer
                            Set Footer = rngField.Sections(1).Footers(wdHeaderFooterPrimary).Range
                            Footer.Select
                            ' Clear any existing text
                            Footer.Delete
                            ' Set up the table
                            Footer.Tables.Add Range:=Footer, NumRows:=1, _
                                NumColumns:=2, DefaultTableBehavior:=wdWord9TableBehavior, AutoFitBehavior:=wdAutoFitFixed
                            With Footer.Tables(1)
                                .Borders.Enable = False
                                If .Style <> "Table Grid" Then
                                    .Style = "Table Grid"
                                End If
                                .ApplyStyleHeadingRows = False
                                .ApplyStyleLastRow = False
                                .ApplyStyleFirstColumn = False
                                .ApplyStyleLastColumn = False
                                .ApplyStyleRowBands = False
                                .ApplyStyleColumnBands = False
                            End With
                            ' Left column
                            Set Selection = Footer.Tables(1).Cell(1, 1).Range
                            Selection.Select
                            Selection.Collapse wdCollapseStart
                            Selection.Text = "Mobiso " + AdminTypes(N) + " Administrator's Guide"
                            ' Right column
                            Selection.Start = Footer.Tables(1).Cell(1, 2).Range.Start
                            Selection.Select
                            Selection.Collapse wdCollapseStart
                            Selection.Text = arrBookmarks(J) + " <field> - <page>"
                            Selection.Find.Text = "<field>"
                            If Selection.Find.Execute Then
                                Selection.Select
                                S = "SEQ Chapter \c"
                                If arrBookmarks(J) = "Appendix" Then
                                    S = S + " \* ALPHABETIC"
                                Else
                                    S = S + " \* ARABIC"
                                End If
                                Selection.Fields.Add Range:=Selection, Type:=wdFieldEmpty, Text:=S, PreserveFormatting:=False
                            End If
                            Selection.Find.Text = "<page>"
                            If Selection.Find.Execute Then
                                Selection.Select
                                Selection.Fields.Add Range:=Selection, Type:=wdFieldEmpty, Text:="PAGE", PreserveFormatting:=False
                            End If
                            Footer.Tables(1).Cell(1, 2).Range.ParagraphFormat.Alignment = wdAlignParagraphRight
                            LastFooterType = arrBookmarks(J)
                        End If
                        ' Insert the section break
                        rngField.InsertBreak wdSectionBreakNextPage
                    End If
                End If
            End If
        Next fldPtr
    Next J
           
    Set rngTemp = ActiveDocument.Bookmarks("Appendix").Range
    rngTemp.Select
    rngTemp.Find.Execute FindText:="^b", ReplaceWith:="", Replace:=wdReplaceOne, Forward:=False
           
    ' Update table of contents and index
    ActiveDocument.TablesOfContents.Item(1).Update
    ActiveDocument.Indexes.Item(1).Update
    
    'MsgBox "Exporting " + AdminTypes(N) + " manual to PDF (" + D + AdminTypes(N) + ".pdf)"
    ActiveDocument.ExportAsFixedFormat D + AdminTypes(N) + ".pdf", wdExportFormatPDF, False, wdExportOptimizeForPrint, wdExportAllDocument
    
    ' Remove the inserted section breaks
    For J = JL To JU
        Set rngTemp = ActiveDocument.Bookmarks(arrBookmarks(J)).Range
        rngTemp.Find.Execute FindText:="^b", ReplaceWith:="", Replace:=wdReplaceAll
    Next J
    
    ' Helpful if you want to build one PDF, then check it
    'If (MsgBox("Built " + AdminTypes(N) + " PDF", vbOKCancel, "Continue?") = vbCancel) Then Exit For
      
Next

MsgBox "Done - Updated PDFs are in " + D

End Sub


This code has good examples of the following with VBA:

  • Set a FIELD tag
  • Delete a section break
  • Export a Word document
  • Create a footer
  • Create a table
  • Insert a section break

This post courtesy of http://mobiso.com

Mozilla/4.0 (compatible;)

This user agent was in the middle of many page requests in my Apache logs, requesting content referenced by link tags in the head section.

After a bit of research on one of the link tag URLs, I ran this script:

IPS=`grep Author access_log | cut -f 1 -d ' '  | sort | uniq`
for IP in $IPS
do
        echo Testing "$IP"
        host "$IP"  
done

In almost every case, the requests came from large organizations - corporations, government agencies, and the military.

These institutions often use proxy servers, and Mozilla/4.0 (compatible;) must be a common user agent setting for the proxy server requests.

In the one case where it wasn’t a large organization, it was a blacklisted IP, and the user agent was Java.

The sample set was limited, but the pattern was clear.

Serializing Data to Pass between Perl and PHP

The objective of this task was to determine if data serialized by PHP could be decoded by Perl.

The first step was to create some serialized data.

In this case, the data is being used to define an interface. An associative array was used, with the first element serving to identify the type of data, and the second to contain the details of the interface. The details is an associative array where each element includes a validation string, label, help or error string, default value, and entered value. This could be extended to include i18n and l10n information, as well as a wide variety of other data.

The PHP code serializes the array, echos it, and then does a var_dump.

<?php
$aData=array(
'type'=>'Magic',
'details'=>array(
'url'=>array(
        'validation'=>'/^[\.\w\-]{1,255}$/',
        'label'=>'URL',
        'help'=>'Valid URL is letters, digits, dashes, periods',
        'default'=>'http://default.com',
        'value'=>'http://domain.com'),
'authid'=>array(
        'validation'=>'/^[\.\w\-]{1,255}$/',
        'label'=>'AuthId',
        'help'=>'Valid Id is letters, digits, dashes, periods',
        'default'=>'',
        'value'=>'')
));
$sSerialized=serialize($aData);
echo $sSerialized.PHP_EOL;
var_dump(unserialize($sSerialized));
echo PHP_EOL;

I took the serialized data echoed by PHP and pasted it into a Perl script.

It uses the PHP::Serialization module to unserialize the data. The code posted here is based on http://www.ohmpie.com/serialization, although this is a more limited example, the ohmpie.com page offers serveral differ serialization approaches.

The printAll method prints all the attributes and values for the class. Note that the values can be reached directly through the object.


#!/usr/bin/perl
# Thanks to: http://www.ohmpie.com/serialization/
use strict;
use PHP::Serialization;
use TestClass;
my $encoded='a:2:{s:4:"type";s:8:"Magic";s:7:"details";a:2:{s:3:"url";a:5:{s:10:"validation";s:19:"/^[\.\w\-]{1,255}$/";s:5:"label";s:3:"URL";s:4:"help";s:45:"Valid URL is letters, digits, dashes, periods";s:7:"default";s:18:"http://default.com";s:5:"value";s:17:"http://domain.com";}s:6:"authid";a:5:{s:10:"validation";s:19:"/^[\.\w\-]{1,255}$/";s:5:"label";s:6:"AuthId";s:4:"help";s:44:"Valid Id is letters, digits, dashes, periods";s:7:"default";s:0:"";s:5:"value";s:0:"";}}}';
my $data = PHP::Serialization::unserialize($encoded);
bless($data,'TestClass');
$data->printAll;

print "URL: ".$data->{'details'}->{'url'}->{'value'}."\n";

print "\n";

This is the TestClass package or module. It only includes the top two elements, type and details, PHP::serialize populates the object with the unserialize call.


#!/usr/bin/perl
# Thanks to: http://www.ohmpie.com/serialization/
#       http://www.perlhowto.com/iterate_through_a_hash
#       http://perl.about.com/od/packagesmodules/qt/perlcpan.htm
#       http://search.cpan.org/~bobtfish/PHP-Serialization-0.34/lib/PHP/Serialization.pm
package TestClass;
use strict; 

#The Constructor
sub new {

        my $obj = {
                type => undef,
                details => undef };
        bless($obj);

        return $obj;
}

sub printAll {
        my $key=undef;
        my %hash=undef;
        my $innerkey=undef;
        my %innerhash=undef;
        my $self=shift;
        my $value=undef;
        my $innervalue=undef;
        print "Type: " .
        $self->{'type'}."\n";
        %hash=%{$self->{'details'}};
        while (($key,$value) = each %hash )
        {
                print "key: $key\n";
                %innerhash = %{$value};
                while (($innerkey,$innervalue) = each %innerhash )
                {
                        print "\t$innerkey: $innervalue\n";
                }
        }
        print "\n";
}

1;

This approach allows data to be stored serialized in a database and read and updated by either Perl or PHP. The structure of the data can change, but the database schema would remain the same.

HTTP Blacklist - Http:BL PHP Code - Generic

This is a generic PHP script that can be used with Http:BL. Http:BL can be used to block requests to a web site based on the IP address. There are several configuration settings that allow you to adjust the performance. In the code below, any IP address identified as suspicious by Project Honey Pot, active within the past 30 days, or with a threat score 100 or greater is blocked.

The easiest way to use it is to include it into the top level of the application, for example:

require_once 'bl.php';

This code just logs the requests and the scores. Once you’re comfortable with it, you can use it to redirect unwanted visitors to a 403 page, or down the rabbit hole.


<?php
/*
abcdefghijkl.2.1.9.127.dnsbl.httpbl.org

Response:
Octet 1: 127 or indicates error
Octet 2: # of days since last activity
Octet 3: Threat score (0=No threat, 255=Extreme threat)
Octet 4: Visitor type
*/

define ('httpBL_API_key','!-- YOUR KEY HERE --!');
define ('httpBL_URL','dnsbl.httpbl.org');
 
/* These are the settings which control which visitors are blocked */
define ('DAYS_SINCE_LAST_ACTIVITY',30);  /* Active within this many days prior will be blocked */
define ('MAX_THREAT_SCORE',100);         /* Anything over this threat score will be blocked */
define ('MAX_TYPE_VALUE',1);             /* Type of visitor - this isn't really bitmapped */
define ('VISITOR_MAP',3);
 
$aOctetMap=array(
'127'=>0,
'DAYS_SINCE_LAST_ACTIVITY'=>1,
'MAX_THREAT_SCORE'=>2,
'VISITOR_MAP'=>3
);
 
$aVisitorType=array(
0=>'Search Engine',
1=>'Suspicious',
2=>'Harvester',
4=>'Comment Spammer',
8=>'[Reserved for Future Use]',
16=>'[Reserved for Future Use]',
32=>'[Reserved for Future Use]',
64=>'[Reserved for Future Use]',
128=>'[Reserved for Future Use]'
);
         
$aSearchEngineSerials=array(
0=>'Undocumented',
1=>'AltaVista',
2=>'Ask',
3=>'Baidu',
4=>'Excite',
5=>'Google',
6=>'Looksmart',
7=>'Lycos',
8=>'MSN',
9=>'Yahoo',
10=>'Cuil',
11=>'InfoSeek',
12=>'Miscellaneous'
);
$sBL=httpBL($_SERVER['REMOTE_ADDR']);
if ($sBL!==null) 
        /* Write out the information to a text file so you can see what is happening */
        file_put_contents('output.txt',$_SERVER['REMOTE_ADDR'].' '.$sBL.PHP_EOL,FILE_APPEND);
        /* Once you are comfortable with your code and settings, you can redirect unwanted visitors elsewhere */
         
function httpBL($sIP)
{
        global $aOctetMap;

        $sOctets=implode('.',array_reverse(explode('.',$sIP)));
        $sURL=httpBL_API_key.'.'.$sOctets.'.'.httpBL_URL;
        $aResult=dns_get_record($sURL,DNS_A);
        if (isset($aResult[0]) && isset($aResult[0]['ip']))
        {
                $aResultOctet=explode('.',$sResult=$aResult[0]['ip']);
                if ((int)$aResultOctet[$aOctetMap['VISITOR_MAP']]<MAX_TYPE_VALUE) return null;
                if ((int)$aResultOctet[$aOctetMap['MAX_THREAT_SCORE']]>=MAX_THREAT_SCORE) return $sResult;
                if ((int)$aResultOctet[$aOctetMap['DAYS_SINCE_LAST_ACTIVITY']]<=DAYS_SINCE_LAST_ACTIVITY) return $sResult;
        }
        return null;
}

The advantage of this approach is that after an IP address has been cleared or cleaned up, access is restored without admin action, so blocked addresses aren’t blocked forever, only for a month or so while they are potentially harmful. The .htaccess Allow,Deny configuration can also be used, but it must be manually maintained, by checking the stats frequently and determining the owner and extent of the IP address block.