Oracle Database Bulk Data Loading Manual

Transcription

1 Oracle Database Bulk Data Loading Manual 1 CONTENTS PAGE Page View 2. Quick Overview for using Oracle 4. Finishing off excel spreadsheet and preparing it to load Turn spreadsheet into text file 5. Transferring text files onto TIGER (SSH shell system) 7. Start up Exceed/Putty Log onto Unix session Unix user commands directories Unix user commands file manipulation 7. Emacs user commands 10. What files should contain and what they are used for. File editing commands 12. Emacs hints SQLPLUS commands 13. Logging out of TIGER

2 Quick Overview for turning a spreadsheet into a new Oracle table 2 Finalise the data within Excel Create a text file from the Excel file and close excel Log-in to tiger Start up Exceed: Start Menu>Programs>Hummingbird connectivity>exceed>exceed (if program on hard drive) Or Start Menu>General Applications>Internet>Exceed Start up Putty: Start Menu>General Applications>Internet>Putty Log-in (as GBHDB) with password NB. CASE SENSATIVE. Create or move to a directory for the table on tiger Decide which section of the database the table should be in, and move to that directory, create one if necessary. Transfer the text file to tiger Go to SSH secure shell client program Change to the correct drive and pc directory containing text file required. Move to the correct directory on tiger Transfer all required text files one at a time from Excel onto database Click on close connection and close the window. Operating System commands on tiger Start up Emacs (text editor) to edit files Create/copy the other files needed to create the table and edit. To leave Emacs Go to File>Exit Emacs (save any un-saved buffers)

3 Working with SQLPLUS (Oracle) on tiger 3 Starting up sqlplus (Oracle) Type sqlplus at the command line Log-in as gbhdb (NB the Oracle password is not the same) Run.sql command file to load onto To leave sqlplus when successfully loaded: quit Logging-out of tiger Type exit This will not work if another program is running It will work whatever directory you are in It must be in lower case letters Close the Putty window Close the Exceed program

4 More Detailed Guide to using Unix and Oracle 4 Stages in turning a spreadsheet into a new Oracle table Finalise the data within Excel Make sure all check-sums are correct Keep place-names as they appear in original Ensure no empty cells other than intended ones If 2 text files have same data in columns add together with extra column denoting difference e.g. gender column If 2 text files have same data in rows and columns add together and change columns headings for differing variable e.g. gender, 0_1, 0_1 becomes M_0_1, F_0_1. To test if columns/rows match: =EXACT (column1_name, column2_name) <ENTER> To test if columns check sums all zero: =MAX(cell1:cell2), <ENTER> =MIN(cell1:cell2) <ENTER> Save a copy of the Excel version of the file Create a text file from the Excel file If there are any commas within the number columns, turn them off Remove all check-sum columns Remove the header rows If the last column of data is sometimes empty, add another column containing XXXX in every row Save the worksheet containing the data as a tab delimited text file i.e. <filename.txt> Close the workbook, WITHOUT saving it

5 Transfer the text file to tiger Tiger Database Manual 5 Use the SSH file transfer software on the desktop (instructions underneath). Go into SSH by clicking on the SSH Secure Shell Client Click on the Quick Connect button Type in the host name: tiger You will need to log-in as GBHDB, or other required database name Click OK and give the password. Click on New File Transfer Button. A window will appear. Click the ASCI button at the top of the window to ensure your files transfer correctly. Move to the correct directory on tiger where you wish to work You should see your PC file structure on the left, and your database file structure on the right. To transfer the file from the PC into the database Select UPLOAD arrow A new window with the PC drives on will appear open the correct folder and select the desired.txt file Click on the UPLOAD button on this screen. The file will transfer Repeat as many times as is necessary. Alternatively: Select file in PC directory on left and drag and drop onto database directory on right. To transfer the file from the database to the PC Select file to transfer on the server Select DOWNLOAD arrow A new window with the PC drives on will appear. Open the folder that you wish to put the selected file in. Click on the DOWNLOAD button on this screen. The file will transfer Repeat as many times as is necessary. If there are no more files to be transferred, click on the Disconnect button and then click Yes. The original window will remain open until you shut it and allows access for reconnection if required.

6 Operating System commands on tiger 6 Logging-in Start up Exceed: Start Menu>Programs>Hummingbird connectivity>exceed>exceed (NB this does not appear to do much) Start up Putty: Start Menu>General Applications>Internet>Putty Log onto Unix server required Log-in (as GBHDB or using account you require) You are now in the home directory! Moving to a particular directory Move to a directory within the current directory: cd <directory name> Move to the directory above the working directory (ie go up one level): cd.. Move to the home (log-in) directory: cd ~ Move to a directory two levels up then one down: cd../../<directory name> ~ Represents the home directory. Creating or deleting a directory If new section is needed, create directory with same name as the table (will be empty) Creating a new directory within the current directory: mkdir <directory name> Deleting a directory: rmdir <directory name> NB this works only if all files in the directory have already been deleted Seeing where you are and what files are there Show the full name of the current working directory: pwd List the files and directories within the working directory: ls List the files and directories in another directory: ls <full directory name>

7 Moving (re-naming), copying or deleting a file To move (or re-name) a file within a directory: 7 mv <current filename> <new filename> To move (or re-name) a file between directories, be in the directory you want to move the file to: mv ~/<dir_name>/<dir_name>/<current filename> <new filename> To copy a file: cp <current filename> <new filename> To delete a file: rm <current filename> To delete all old files in a directory: rm *~ NEVER USE COMMAND: rm *!!! It will delete all files permanently!!! Each table must be in a relevant directory and have a.dat file (actual data from text file).sql file (command file for table) _doc.sql file (creates table documentation on website, template in home directory).ctl file (controls initial data loading).log file and.bad file (automatically created when loading onto database.) Create.ctl, and.sql by copying similar files and amend as necessary. Create doc.sql by copying template_doc.sql in home directory. Up arrow will copy the previous command line in unix onto present one. Remove all tabs and replace with : Tr \t <current filename >new filename Working with Emacs / Text editor Start up Emacs (the text editor): Type Emacs & <ENTER>

8 Loading and unloading files Tiger Database Manual 8 To open a file in a new buffer: File>Open File and then type the file name at the bottom of the screen To save the buffer as a new version of the original file: File>Save Buffer To save the buffer as a new file: File>Save Buffer as type in new filename) Panic key! If you make a mistake in Emacs and want it to stop something, try typing <CTRL>-G it will often work! If this does not work, and you are left with something you did not want, remember that all you have changed is the buffer, not the file so just avoid saving it and close the buffer then reopen file and start again! Moving around Use the mouse and arrow keys (!), or: Go down a screen: <ESC>, <CTRL>+V (i.e. hold down CTRL, type V) Go up a screen: <ESC>, V (i.e. type ESC, then type V) To go to the start of the buffer: <ESC>, <SHIFT> + < (i.e. type ESC, then hold down shift and type <) To go to the end of the buffer: <ESC>, <SHIFT> + > (i.e. type ESC, then hold down shift and type >) To move to another buffer: select if from the buffers menu Changing text The simple method is to move somewhere, then start typing or use the <BACKSPACE> key! Global replace: Type <ESC>, <Ctrl>+X (which brings up an M-x prompt at the bottom of the Emacs window Simple global replace: Type replace-string (or repl<tab>s<tab>) at the M-x prompt, then <RETURN> Type the text you want to replace, wherever it appears in the file, followed by <RETURN>

9 Type the text you want to replace it with, followed by <RETURN> 9 Emacs goes away and makes the changes! Global replace using Regular Expressions (i.e. wild cards) <ESC>, <Ctrl>+X, repl <TAB>, r <TAB>, *.<word to be replaced>, <ENTER>, <new word> <ENTER> Global search: <ESC>, <CTRL>+<ALT>+S, <ENTER> type in text looking for. It will find relevant strings of letters as typed. Query Replace, to confirm each change as it appears: <ESC>, <Ctrl>+X, que <TAB> <ENTER> Standard text groupings need vertical bars either side E.g. repl<tab> s <TAB> ALL

10 Delete text: Tiger Database Manual 10 Type <ESC>, <CTRL>-K (i.e. hold down CTRL and type K) at the beginning of line to be deleted. Keep going until all lines to be removed have gone from screen (NB. This acts like cut function in excel) Replace deleted text: Type <ESC>, <CTRL>-Y (i.e. hold down CTRL and type Y) at the beginning of where you want to place the text and it will reappear. This can be repeated as many times as desired (NB this acts like the paste function in excel) Ensure the buffer ends at end of data and nothing is after it. Save buffer by <ESC>, <CTRL> + X, S, <ENTER> Anything typed next to rem or inside < > in the file is invisible to Oracle. All text must have marks. Must be two of them (i.e. at each end of the text to appear on web) otherwise text will stop halfway through. Creating column using data Contents of files Control (.ctl) file List of columns must match content and order of columns in the.dat file exactly or will not work. Must have just, at end of each line as this puts columns into right order. Data (.dat) file After imported from excel as.txt change name, to <file name>.dat Change all <TAB> to If appropriate remove all and and, replace with Remove any spaces from between If last row is XXXX replace with Use replace regular expression command to replace <space> at the end of lines <ESC>, <CTRL>+X, Repl <TAB> r <TAB> $ <ENTER> <ENTER>

11 Start of each line replace with regular expression so all the same. 11 E.g. Repl <TAB> r <TAB> ^ALL <ENTER> ALL <ENTER> Documentation in SQL language (doc.sql) file All typing done next to rem and between <> is invisible All typing to appear on page must have marks. If inverted comma needed in the text use two together so text not cut off. Ensure give full source table name, volume and year of census. One line description must be short to id information in section only. SQL language (.sql) file. Column names do not have to be in same order as data file. Column names should be desired display order and end with ); Must have commit at the end of each command. Create columns using data i.e. update <table_name> set new <column name> <column X> + <column Y>; Multiply columns in this file, ensure any sums done first! E.g. update <table name> set <column 1> = <column 1> *10, <column 2> = <column 2> * 10; Text string variable should be set at very long limit to begin with. Once this has been loaded then check for the longest string length. Select max (length(variable)) Once identified change the long limit to one just beyond the longest string selected otherwise the index will not build. Create an index at the end of the.sql file before granting public access. Create unique index <table name>_idx1 on <table name> (civ_par, lg_dist, lg_type, adm_cnty, ward); Do this for as many indexes as deemed necessary just changing the index number and the variables included. E.g. Create index <table anme>_idx3 on <table name> (lg_dist, lg_type, adm_cnty);

12 Helpful hints on Emacs $ represents end of a line ^ represents start of a line To leave Emacs Tiger Database Manual 12 Go to File>Exit Emacs (you will be asked to save any un-saved buffers) Working with SQLPLUS (Oracle) on tiger Starting up sqlplus (Oracle) Type sqlplus at the command line Log-in as gbhdb or required database name Type in password. NB the Oracle password is different to the system password) To run a unix system command without leaving sqlplus (e.g.); host ls or host pwd (NB you cannot change directories while in sqlplus) To run a.sql command (NB the.sql at the end is unnecessary Re-type password when prompted Ctrl+C stops the command running or stops the view

13 To see if loaded successfully: host more *.log or host more <filename>.log Press space bar to see all errors To see which rows unsuccessfully loaded: host more *.bad or host more <filename>.bad Will not like null quantities, 0 is fine Error message ORA-0140 means inserted value too large for text column (unlikely to be numbers column) To leave sqlplus, and return to the tiger command prompt: quit Logging-out of tiger Type exit This will not work if you have another program running It will work whatever directory you are in Close the Putty window Close the exceed program. 13