ARIES Flat File Importer

Transcription

1 ARIES Flat File Importer The Flat File Importer is a utility program run outside of ARIES that lets you bring data into an ARIES database from a generic source file. The source can be a plain text file that uses spaces or tabs to separate the data items, or, more typically, a CSV (comma separated values) file that you create from Excel. You specify the instructions to match up this data with your database items in an import template file. The steps needed are: 1. Format the input data you want into a source file. See the discussion later in this document for details of this step. 2. Create or modify an import instruction template to control this data and map it to your database. See the discussion later in this document for details of this step. 3. Open both your source file and the import instructions template together and examine them for compatibility. 4. Run the import utility. This will be discussed first. Running the Importer First check to be sure you have both a properly formatted source data file and an instruction template to control the import. Then run the FlatFile.exe utility by double clicking on either the file name in the ARIES\Programs folder or on its icon, which you have placed on the desktop. You do not have to be in ARIES to do this. The following dialog will appear. Follow these steps: 1. Enter the location and name of the instruction template you will use, or click the Browse button to select it. 2. Enter the location and name of the input source data file you will use, or click the Browse button to select it. 3. Use the Browse button to select the datasource into which you wish to import this data, or you can enter its name. 4. Click Import to begin the process You will next see a DBS selection dialog. Highlight the DBS to which you wish to assign the data and select OK. Enter the ARIES System Owner name if needed.

2 Input Data Source File The input source data can be in a plain text file that uses spaces or tabs to separate the data items, or, more typically, in a CSV (comma separated values) file that you create from Excel using its Save As feature. Excel spreadsheets cannot be imported directly. The file can contain header-type text rows on top, but they must be excluded by including a SKIPLINES = instruction in the import template. The remaining lines in the file must each contain: The same set of items in the same order, as specified in the data lines section of the import template. This is what makes the source a flat file. The value that is specified on the LOOKUP = instruction in the import template. This defines what the property is. A key to successful imports is ensuring that the fields in the source data file correspond to those specified in the instruction (.imp) template. Formatting Excel Data for Importing You may have, or receive, spreadsheets that you want to import that contain blank cells or varying formats. You can generally reformat and save them as.csv files so they can be imported properly. Several example files are provided in the ARIES\Programs folder: FlatFile Example.xls (original spreadsheet data), FlatFile Example.csv (converted comma separated source data), and FlatFile Sample.imp (import instruction template). To start, open and view your source data file in Excel. 1. Look for blank or partially blank columns or areas. Generally you should delete any unneeded columns (you might want to first make a copy of the whole spreadsheet). Use the fill down feature to copy repetitive information into any blank areas, shown as shaded in the example below. For some items, a blank cell is appropriate, such as having no water production for April, and should be left blank. 2. Either remove header rows in Excel, or be sure that you are using the SKIPLINES = instruction in the template. In the example above, the first line should be removed, or SKIPLINES = 1 should be included in the import template.

3 The original spreadsheet file; blank cells to be filled are shaded. 3. If the Excel dates are non-standard either reformat them by specifying a custom format type of MM/DD/YYYY, or include a DATE = instruction line in the template. In the case above, the dates are not 10-character and are at the first of the month. Reformat them in Excel by highlighting the date column, right clicking and selecting Format cells, and entering the desired mm/dd/yyyy format under Type, as shown below.

4 This puts the dates in standard format in the resulting.csv file, but at the first of the month. So also include the instruction DATE = STDLAST in the import template to shift the day to the end of the month during import. 4. If the data includes API numbers, Excel may convert them to scientific notation, due to their length. Reformat them to text or to custom 10- or 14-digit numbers using ########## or ##############. Note that.csv files will not retain this custom number formatting, but will retain the values. 5. Use the Save As feature to save a copy of the spreadsheet. Under file type select one of the.csv styles. You should respond OK to the message that appears informing you that special formatting will not be saved. 6. Open the resulting.csv file in Notepad or other text editor to ensure that the formatting is what you want. Do not open and then save this file using Excel because it will convert it back into a non-importable spreadsheet. The resulting.csv file Import Instruction Template This is a plain text file you create or modify in Notepad or other text editor. It tells the importer how the data should be placed into your ARIES database. It should have the file name extension of.imp so the Browse feature can easily display a targeted list. This file specifies:

5 How to match the data with your properties Which tables and items within ARIES you will be updating with which source data values How to treat some of the values, such as dates in various formats Other special methods The following example template illustrates the required and optional instructions. TABLE=M TABLE=MP ; LOOKUP = M.API DATE = MM/LL/YYYY DELIMITER = COMMA CONST = M.RES_CAT = 1PDP ; DEFAULT = M.MAJOR = GAS ; SKIPLINES = 1 ; CREATE = NO ; FACTOR = MP.OIL = 1000 ; ; import these items M.LEASE = 3 M.STATE = 2 M.MAJOR = 6 M.WELL_ID = 4 M.API = 1 M.OPERATOR = 5 MP.P_DATE = 7 MP.OIL = 8 MP.GAS = 9 MP.WATER = 10 TABLE Instructions The two TABLE = Alias Name instruction lines above specify that you will be importing into both the M (the alias for Master) table and the MP (the alias for the monthly production) table. You must have at least one and at most two TABLE lines, and the TABLE = line(s) instruction must be at the top of the file. Note that the equal signs do not have to align, and except for TABLE the items do not have to be in any specific order. You can enter a semicolon in column one to indicate that the line is a comment and is not to be processed by the utility. It can also be used to turn a particular command on or off. LOOKUP Instruction A LOOKUP instruction line is required to specify the value that is common to both the source file and your database that is to be used to identify each property uniquely. It is used to import the data into the

6 correct property or to create a new property if one does not yet exist. Examples for lookup values could be M.API, M.WELLCODE, or others. You must have a LOOKUP instruction and this item must be one of those defined for importing below. Additional Instruction Lines Many other instruction lines are provided so you can manipulate the incoming data. DATE = The standard importer treatment for dates is to treat them as 10-character MM/DD/YYYY format and import them as they are, for example 03/23/2012. Though not necessary, entering DATE = STD produces this standard treatment. If your source data is in a different format, or you want to shift the day value, include a DATE = instruction line using one of the following values. If your source file uses a non-standard date format, specify what it is by entering YYYYMMDD, YYMMDD, DD-MON-YYYY, or MON-YYYYLL (which both reformats the date and shifts to the last day of the month) To convert standard format source dates that have various day values to the last day of the month, use STDLAST or MM/LL/YYYY. Note that the MP table normally uses the last day of the month. To convert standard format source dates that have various day values to the first day of the month, use STDFIRST. This is rarely used. SKIPLINES = You can skip lines at the top of your source file, such as column headers, by including this instruction to specify the number of lines to skip. For example SKIPLINES = 2 will skip the first two lines. CREATE = If source data exists for a property not yet in your database based on the LOOKUP item, the importer will create a new property by default. If you do not want new properties created, include CREATE = NO in your instructions. The only other value you can enter is YES, which is the same as omitting or commenting out a CREATE = NO instruction. DELIMITER = If your source data file has the values on each line separated (delimited) by something other than a comma, you must include the DELIMITER = instruction. For example, if your source file is delimited by tabs, include DELIMITER = TAB in your instructions. The choices include: COMMA commas, the default TAB tabs SPACE one or more spaces. Note that imbedded spaces that exist within text items in your source will be treated as delimiters unless you enclose items within quote marks. char enter any character, enclosed in quotes. For example you can enter ; to specify that semicolons are the delimiters. Note that which quote symbol you can enter here (single or double quote marks) depends on the current QUOTE = instruction.

7 FACTOR = TABLE.COLUMN = If you need to apply a factor to a particular source data item, include a FACTOR = instruction for it. For example, to convert incoming oil production volumes that are in Mbbl to the standard database units of bbl, enter FACTOR = MP.OIL = CONST = TABLE.COLUMN = If you want to populate one of your database items with a constant value for each property imported, regardless of what is in the source file, include the CONST = instruction. For example, to populate a partner name you could include CONST = M.PARTNER = My Partner Company. Note that quote marks are needed around phrases containing embedded spaces. The constant can also be a number. DEFAULT = TABLE.COLUMN = If you want to populate one of your database items with a constant value ONLY when that item is missing from the source file, include the DEFAULT = instruction. For example, to populate the value for county code if it is not present in the source file, you could include DEFAULT = M.CCODE = 16. IGNORE = If you do not want to import some of the data lines from the source file, include this instruction followed by the initial characters of the line you want to ignore. For example IGNORE = DEF234 will prevent the importing of all source data lines beginning with those characters, regardless of what follows on the line. You can enter multiple IGNORE = instructions. You can also skip a block of source file lines. For example to ignore the line above plus the 11 lines that follow it, enter IGNORE+11 = DEF234. If you need to match text that has leading or embedded blanks, enclose it in quote marks. For example, you could specify IGNORE+11 = DEF234 TX HUNT. This string must be exact, including the proper number of blank spaces in front of or between characters. QUOTE = The standard indicator expected for grouping phrases together is the double quote mark, for example Red Horse A-3. If the source file uses this mark as part of the data, you can change the importer to recognize the single quote mark for grouping phrases instead, by including the QUOTE = instruction. You can switch back and forth between these at different locations within the template by entering multiple QUOTE lines. TRIMSPACE = If you want to remove leading, trailing, or multiple embedded blank spaces from text you import, include the instruction TRIMSPACE = YES. PIVOT_YEAR = If the source data uses two-digit years, the importer assumes 1900 for the century if the year is 30 or greater. If the year is less than 30 it assumes So 29 becomes 2029 and 31 becomes If you want to change this pivot year, enter the PIVOT_YEAR = instruction. For example, to import source

8 production data from the early 1900 s, you could enter PIVOT_YEAR = But note that this would also change treatment of data during Data Item Lines The subsequent lines in the above example specify the items in your database that are to be imported and the columns within the source file where these values are located. You indicate the items using the Table Alias (dot) Item Name format. For example, M.LEASE indicates the LEASE item within the M table. You do not have to import all of the columns. The numbers following the equal signs specify which column in the source file contains the wanted data. For example, M.LEASE = 3 indicates that the item is located in the third column or section of the source data lines, as separated by the specified delimiters AC