No More Dual

Ok, that's Duel rather than Dual (and only oldies will get it anyway). Moving on...

Since version 23c Oracle has made it no longer necessary to do this:

SELECT SYSDATE
FROM DUAL;

Instead you can just do this:

SELECT SYSDATE;

This can look a little bit strange till you get used to it but the above queries are equivalent. A couple of other examples without FROM DUAL:

SELECT 1
UNION ALL
SELECT 2;

SELECT LEVEL CONNECT BY ROWNUM <= 4;

Direct Joins in Update and Delete Statements

You can now (v23ai/26ai) directly join tables within updates and deletes. These are more readable than using subqueries:

UPDATE EMP E
SET E.SAL = E.SAL + (E.SAL * .1)
FROM DEPT D
WHERE E.DEPTNO = D.DEPTNO
  AND D.LOC = 'YORKSHIRE';

Or this:

DELETE FROM EMP E
FROM DEPT D
WHERE E.DEPTNO = D.DEPTNO
  AND D.DNAME = 'RESEARCH';

Insert Improvements

The SET clause - INSERT statements like UPDATE statements...

The following:

INSERT INTO CLAIMS (ID, CREATED_DT, CREATED_BY, RECEIVED_DT, STATUS)
VALUES
(CLAIM_SEQ.NEXTVAL, SYSDATE, USER, :1, :2);

Can be replaced by:

INSERT INTO CLAIMS 
SET ID = CLAIM_SEQ.NEXTVAL, CREATED_DT=SYDATE, CREATED_BY=USER RECEIVED_DT=:1, STATUS=:2;

The SET clause is much easier for column to value alignment making them more readable and far less likely to set any columns to the wrong value.

Oracle also supports multirow inserts like these:

INSERT INTO CLAIMS (ID, CREATED_DT, CREATED_BY, RECEIVED_DT, STATUS)
VALUES
(CLAIM_SEQ.NEXTVAL, SYSDATE, USER, :1, :2),
(CLAIM_SEQ.NEXTVAL, SYSDATE, USER, :3, :4);

And (with SET):

INSERT INTO CLAIMS 
SET (ID = CLAIM_SEQ.NEXTVAL, CREATED_DT=SYDATE, CREATED_BY=USER RECEIVED_DT=:1, STATUS=:2), (ID = CLAIM_SEQ.NEXTVAL, CREATED_DT=SYDATE, CREATED_BY=USER RECEIVED_DT=:3, STATUS=:4);

Note the use of parenthesis here to divide the rows.

I was curious to see if it was possible to use the returning clause returning into a collection e.g. RETURNING ID BULK COLLECT INTO r_ids; but this was not possible for either of the above inserts and resulting in the following errors respectively:

ORA-63853: RETURNING clause is not allowed with a multi-row INSERT INTO SET clause.

ORA-63809: returning clause is not allowed with INSERT and Table Value Constructor

Follow my blog and YouTube Channel for more Oracle coding tips.

Back to the Future

I recently had a week off work, when I came back Github Copilot had arrived. Two days later we had upgraded to Apex 26.4 on our development environment. I have to say it is a very exciting time and it feels for me to be an important precipice where there is so much to make use of and discover. There have always been blockers and effort involved in upgrading the toolset but with Oracle Apex and Autonomous databases these are minimised so the benefits can be experienced without the associated rigmarole.

Initial Observations on the Use of AI

I have had recent experiences of searching for coding solutions via a search engine and have had reasonable success in some cases. I have encountered some things for the first time:

• AI providing javascript commands to be run on the command line in order to debug why something is failing. AI can be very confident on prescribing solutions though and I have witnessed upwards of ten false observations. (This is happening because of x, y, z - run this script to prove this...) - wrong.

• Code completion that does language corrections (expected) but also provided meaningful descriptive text based on a code and some associated letter text. This behaviour and similar behaviours will improve the solution being offered and make me wonder if this can streamline the requirements gathering process.

• Installing software on my behalf (it did ask me first!)

• Creating a new Apex page based on some prompts... Not just the page but supporting objects such as a view and a package. It even checked the application to adhere to existing patterns in our application. It didn't work flawlessly but was a great starting point.

The use of Github Copilot has impressed me - not only does it create pages and components it also inspects the apex application to make sure the solution adheres to existing patterns! When trying to add Oracle Apex and DB skills it was even uploading software onto my laptop.

How Good Might AI Get?

Imagine having access to a team of the best developers in the world at your fingertips. As coders we often work alone (peer reviews aside) and some of us can be considered experts in specific areas. What if we were experts in everything? What if we could automatically access every relevant reference and test our code as it was written. Multiply this by a million.

If We're not Coding then what are we going to be doing?

Although AI is going to outshine human coders it is only going to be able to make the solution we tell it to. Therefore our skills are less about typing every line of code and manually building screens but more about being good at telling AI exactly what is required and iteratively checking the result. At the moment I've been using AI to provide a starting point and then finishing this solution off manually, as I go round this process a number of times I will have built up some guidelines and possibly a library of prompts. It is also doable to provide coding standards that AI must follow. Basically we're the human in the loop, we have to build into our prompts guidance so that AI cares about the outcome.

Exciting Times

Another reason why right now is so exciting is that Oracle Apex has just become AI enabled but the models using it are currently lacking in specific skills and knowledge. As this learning happens we are going to see an exponential growth in the abilities of these coding agents.

What Will Happen When Users are No Longer Human?

As applications can be written by AI it seems likely that AI be writing programs to be used by AI. I say programs rather than applications as there may not be a front end at that point other than a limited application for humans to look at data/reporting and perhaps admin screens to govern the AI functionality. There could even be functionality created as it is needed. As a simple example - AI encounters some data items that are as yet not captured in the system. The relevant agents amend the back end tables and amend the relevant packages so that the new data can be added. This could happen with barely skipping a beat.

Whilst the OCI client was being set up I worked on the solution which included code to do the following:

• An initial load of client data from a spreadsheet to store the documents against.

• Get documents from the object storage staging area.

• Load them into the DB into a documents staging table as blobs/filenames etc. on the DB.

• Reconcile the data (needed due to a previous load of some of the files/client data).

• Load the documents to object storage storing them in the application documents table (docs named as a uuid held in data).

• Delete the blobs from the staging tables.

I developed the solution and loaded a sample of documents against a small no. of clients as a proof of the solution. To do this I loaded files into the bucket manually via the OCI web application. Once this was done I thought I'd cracked it...

However, when I first tried to run the package procedure to load the full list of documents into the document staging table I found that it did not run in a timely manner. I left it running for 6 hours before killing the process...

This was my original program unit:

procedure load_oci_files (p_input_id number) is
begin
    FOR rec IN (
        SELECT *
        FROM
        (
            SELECT object_name, file_name, ROWNUM rn
            FROM
                (
                SELECT object_name /*substr/replace*/ file_name,
                       object_name            
                FROM   DBMS_CLOUD.LIST_OBJECTS(credential_name =>                   
                       'OCI_OBJ_STORAGE',
                       location_uri =>
                       g_base_url||g_ra_micro_bucket)
                WHERE BYTES > 0
                ORDER BY object_name
                )
            WHERE FILE_NAME NOT IN ('lfdNr', 'Thumbs.db')
        )  
        WHERE rn BETWEEN (p_Input_id*10000)-9999 AND p_Input_id*10000 
        )  
    LOOP
        l_Blob := DBMS_CLOUD.GET_OBJECT(
                  credential_name => 'MY_CRED_NAME', -- pre-created credential
                  object_uri      => rec.object_uri
                                        );
  
      insert into DOCUMENT_FILE
        (
        ...
        FSIZE,
        FCONTENT,
        ...
        )
        values
        (
        ...
        dbms_lob.getlength(l_Blob),
        l_Blob,
        ...
        );        
      END LOOP;
end load_oci_files;

Parallel Processing

I looked into how I could speed up the program unit and looked into parallel processing (a first for me). Basically parallel processing allows a task to be split and run in parallel by a number of 'workers'. It is generally table based and works by splitting the load down by either row_id or a number (e.g. id column). Once it is set up then the handling of managing the split of ids and parallel processing jobs etc. is managed by Oracle.

I had a ‘conversation’ with AI via a popular search engine which recommended the code. I reworked this into my package whilst also verifying and learning from this page: DBMS_PARALLEL_EXECUTE. I added a separate program unit to load the filenames and URIs into a STAGING_LOAD_DOCS table, to allow the ROWID split, and the remaining code now looked like this:

function load_oci_files_sql (p_input_id number) return clob is
  l_sql CLOB;
BEGIN
  l_sql := q'[
    DECLARE
      l_blob blob;
    BEGIN
      FOR rec IN (
        SELECT input_id, object_uri, file_name, object_name
        FROM STAGING_LOAD_DOCS
        WHERE ROWID BETWEEN :start_id AND :end_id
          AND input_id = ]'||p_input_id||q'[
      ) LOOP
        l_Blob := DBMS_CLOUD.GET_OBJECT(
                      credential_name => 'MY_CRED_NAME', -- pre-created credential
                      object_uri      => rec.object_uri
                                        );
      insert into DOCUMENT_FILE
        (
        ...
        FSIZE,
        FCONTENT,
        ...
        )
        values
        (
        ...
        dbms_lob.getlength(l_Blob),
        l_Blob,
        ...
        );        
      END LOOP;
     
    END;
  ]';

  return l_sql;
end load_oci_files_sql;

procedure create_and_run_load_task(p_input_id number) is
    l_sql CLOB;
BEGIN

    -- Drop if exists to avoid wrong state
    BEGIN
        DBMS_PARALLEL_EXECUTE.DROP_TASK(k_Task_Name);
        EXCEPTION
        WHEN OTHERS THEN NULL; -- Ignore if task doesn't exist
    END;

    DBMS_PARALLEL_EXECUTE.CREATE_TASK(k_Task_Name);

    -- Chunk by ROWID so each chunk processes a subset of rows
    DBMS_PARALLEL_EXECUTE.CREATE_CHUNKS_BY_ROWID(
        task_name   => k_Task_Name,
        table_owner => USER,
        table_name  => k_Task_Name,
        by_row      => TRUE,
        chunk_size  => 1  -- one file per chunk
    );

  l_sql := load_oci_files_sql(p_input_id);

  DBMS_PARALLEL_EXECUTE.RUN_TASK(
    task_name      => k_Task_Name,
    sql_stmt       => l_sql,
    language_flag  => DBMS_SQL.NATIVE,
    parallel_level => 4  -- number of concurrent workers
  );
END create_and_run_load_task;

The key sections in the code are this line in the select statement:

WHERE ROWID BETWEEN :start_id AND :end_id

And also these parameters: for example:
chunk_size => 100 -- 100 files per chunk
parallel_level => 8 -- number of concurrent workers

These are used to split the load into multiple sections. E.g. for this example:

8 workers in parallel will process 100 rows each.

Lessons Learnt and Outstanding Questions

AI suggested a chunk size of 1 which seemed extreme so I asked AI to explain this. The reasoning was that the dbms_cloud.get_object call and inserting the blobs was the most expensive operation. Also it mentioned that this would give the least lag time compared to grouping a set number of different file types.

BUT when I re-examined this whilst writing this blog post then I was given findings to the contrary:

In my case I proceeded with a chunk size of 1.

Upon further reading about DBMS_PARALLEL_EXECUTE I've realised that the examples are primarily for sql update statements (inserts and merges are also mentioned) and that the criteria are focused on this. Indeed the DBMS_PARALLEL_EXECUTE.RUN_TASK parameter is sql_stmt. PL/SQL code can be used though, clearly so I am interested to learn more about this.

Another point is around error handling - the following code (the WHILE loop) is in the examples:

L_status := DBMS_PARALLEL_EXECUTE.TASK_STATUS('mytask'); WHILE(l_try < 2 and L_status != DBMS_PARALLEL_EXECUTE.FINISHED) LOOP 
  L_try := l_try + 1; DBMS_PARALLEL_EXECUTE.RESUME_TASK('mytask');
  L_status := DBMS_PARALLEL_EXECUTE.TASK_STATUS('mytask'); 
END LOOP;

In my case had no such error handling in the code but this was an overseen, one off process of running the scripts for the migration and I was running the following to check for errors (there were none).

SELECT *
FROM   user_parallel_execute_chunks
WHERE  task_name = 'RA_MICRO_LOAD_DOCS'
ORDER BY chunk_id;

In the case of reusable production code then the error handling is likely to be essential.

Use of AI in Coding Solutions

As mentioned earlier, I have been partially assisted by AI (via an internet search engine in this case) in providing this solution. There are some obvious pitfalls to this and likely some non-obvious ones too. In closing this blog post here are some of my initial concerns (although I'm barely scratching the surface - more to come on this topic):

• Not fully understanding the solution provided by AI (in reality this is going to happen regularly).

• AI not understanding the problem, the problem being miscommunicated and/or the solution being incomplete. Solutions should be thoroughly tested.

• With normal, manually typed code the practice of thinking about and writing the code means the code is more likely to be understood by the programmer. AI provided code means there is a layer of separation between the programmer and the solution.

• As the use of AI coding becomes more widespread us coders are inevitably going to have diminished skills in writing code although code reading and diagnostic skills are still a strong requirement currently.

It is therefore sensible to routinely export your working copy application so it can be refreshed from the export... BUT if your working copy has gone and you have not exported then, fear not, you may be able to restore from a backup.

How to Restore Your Working Copy From a Backup

Go to 'Workspace Utilities' > 'All Workspace Utilities'

Go to 'Manage Backups'

Click the working copy you want to restore

For the working copy you want to restore (likely the latest) click the 'Action' drop down and select 'Download'. Note: you cannot 'Restore' the working copy directly as it no longer exists.

Save the zip file then open the file location and extract the file

Go the main application and click 'Create Working Copy'

Give the working copy any name e.g. WC1 - it will be replaced when we restore the working copy. Click 'Create Working Copy'

You will now be in your new working copy. Make a note of the application id - (112 in this example)

In your new working copy select 'Export / Import'

On the following screen select 'Import' and then drag and drop or select the application being restored. Click 'Next'

For 'Import As Application' select 'Change Application ID' and set the 'New Application ID' to match the working copy application id. Click 'Import Application'.

Your working copy should now have been successfully restored from the backup

Introduction

I often think when developing software that the maximum learning happens when there are challenges or misunderstanding with how something works. I have at some points probably made my fair share of mistakes but I always try to learn from them so the same mistake repeated is rare. Here is my account of a recent project which will be useful as a reference for object storage, OCI client and associated coding including using parallel processing but also of interest for the insights of mistakes made and lessons learned – without them being glossed over, the naked truth!

Initial Steps/Configuration

I was tasked with loading files to object storage and then associating with data in our database. The accepted way to achieve this was via the OCI client being installed onto the server where the files resided. Then bulk load commands could be used to load the files from the command line into the OCI staging area. I needed to work with the service desk to get the OCI client installed. The first issue was due to restrictions on the server where the files resided prevented the OCI client from being installed. A copy of the server was generated and then the OCI client was installed.

Once the OCI client software had been installed I then set up the client on the server. I found some instructions that instructed of setting up a new user, group and policy but I reasoned that I could use my own user so I set up the ssh keys against my user and copied the private key to configure the OCI client on the server.

Some initial problems that were encountered with the OCI client were:

1/ OCI was installed but not recognised when running commands via the command line. E.g. oci -i

The fix was to run:

set PATH=%PATH%;C:\OCI\bin

to add c:/OCI/bin to the path environment variable. Note: I could run this command myself (as a non-admin user) but I found I had to rerun it for subsequent log ons in order to use the OCI client.

2/ After the above was resolved the next error was "No Python" at... saying that Python wasn’t installed.

Once the OCI and Python were installed I could run oci commands successfully. Yes!

Bulk Upload Commands

I could then start to run the bulk commands. I initially started with this command from Lee Dixon's excellent blog: Bulk Uploading 25M Files to OCI Object Storage

oci os object bulk-upload -bn "ASCEND_INVOICES" --src-dir "H:\Ascend\M10" --overwrite --parallel-upload-count 500 --object-prefix "H/Ascend/M12/" --exclude *.Log > H_M10.json

In my case the folder structures looked like this:

The folders shown represent years, this was followed by month and days folders as follows:

All of the documents comprising of pdfs, docx and msg files were in the day folders (e.g. '19' in the above screenshot).

My initial command to upload files from the 17 folder was:

oci os object bulk-upload -bn "ra_micro_non_prod" --src-dir "X:\RA\ar\17" --overwrite --parallel-upload-count 500 > ar_17.json

As (I reasoned) that I don’t need to change any paths I omitted the object-prefix parameter completely.

Everything Seemed Rosie

I then checked my resulting log files by navigating to them and checking the counts by searching a string and obtaining the count. (Lee Dixon has a better way of doing this from the command line but I went my own way). I then checked the ‘Approximate’ count on the server to ensure that the expected files were uploaded. Satisfied that the logs seemed to indicate everything had gone across and the OCI server count being approximate I assumed that all the files were loaded. Cue a rather optimistic and criminally low estimate of the remaining work when asked how the project was going by our CTO…

At this point I was feeling buoyant and surprised at how simple it all seemed… But at the time of giving my update I had lost sight certain key facts:

1/ I had done no load testing so didn’t actually know if my current solution would work successfully.

2/ Although there was no indication in the logs not all the files had been loaded (as transpired later).

Silence Isn't Always Golden

The issue I later uncovered was due to an assumption about the path parameter in the bulk load command. As the source directory was e.g. --src-dir "X:\RA\ar\17" then this uploaded to the OCI bucket as follows - starting at that folder point:

Note that as the top level structures weren't uploaded then all of the files from originating year and month folders e.g. 18/01, 19/01, 20/01 etc. were being loaded into the 01 folder... this was in effect silently removing all files with duplicate names. Looking at the bulk load command again:

oci os object bulk-upload -bn "ra_micro_non_prod" --src-dir "X:\RA\ar\17" --overwrite --parallel-upload-count 500 > ar_17.json

The overwrite command means that any duplicates are not logged, only overwritten. On realising this the command became:

oci os object bulk-upload -bn "ra_micro" --src-dir "X:\RA\ar\17" --overwrite --parallel-upload-count 500 --object-prefix "ar/17/"> ar_17.json

Note the object prefix parameter which is essential in this scenario to ensure that duplicates are not filtered out, especially when also using the overwrite command. With the above command the file paths were correctly represented in the OCI bucket (see below) and the file counts matched the source folders.

I used a query similar to this to check the files residing in the staging bucket:

SELECT * FROM DBMS_CLOUD.LIST_OBJECTS( credential_name => 'MY_CRED_NAME', location_uri => 'https://mybucketuri') WHERE BYTES > 0 ORDER BY object_name;

This was checked against counts of files on the server (which I viewed by inspecting the folder properties on the windows server). The counts lined up perfectly now that the files were going into the correct folders.

Please join me, for more posts in this series as they become available.