Group hackathon on interpolation code (Fortran)

Compilation

• Code compiled and run on laptop with gfortran on Ubuntu.
• But not easy for everyone to compile locally.
• Tried on cocal but did not work very well
• Im not so familiar with cocalc
• so he problem is not cocalc

Share the code & development in Github

Github username

• annefou
• maryjanebopape
• pmulovhedzi
• leeseetja
• NohlahlaDlamini

Create a github repository under github.com/annefou

https://github.com/annefou/hackathon-south-africa

Compile and run the code

• Open cocalc.com and create a terminal

  • let us know when you are done.
    • annefou done.
    • leeseetja done
    • NohlahlaDlamini done
    • maryjanebopape done

• console.cloud.google.com seems to be free and we can run

  • open console and clone repository
    • annefou done.
    • leeseetja done.
    • NohlahlaDlamini done

Try out to update the console:

sudo apt-get update
sudo apt-get install -y gcc gfortran

Get the code:

git clone https://github.com/annefou/hackathon-south-africa
cd hackathon-south-africa

Then compile and run the code:

gfortran interpolate_sub.f90 -o interpolate.x

And to run

./interpolate.x

Installation of pFUnit (Fortran unit testing)

You may need to install additional packages (on ubuntu):

sudo apt-get install cmake make python3 m4

And then

cd $HOME
git clone https://github.com/Goddard-Fortran-Ecosystem/pFUnit.git

cd pFUnit
mkdir build
cd build
cmake ..
make
make tests
make install

Test pFUnit with "custom" code

• Take the test from CodeRefinery test lesson: https://coderefinery.github.io/testing/

Let's try an existing Fortran test with pfUnit:

https://coderefinery.github.io/testing/test-design/

We create test-cr.f90 (use any of your favourite editor vi, nano, etc.) with:

module factorial_mod
contains
   ! computes the factorial of n
   integer function factorial(n)
      implicit none
      integer, intent(in) :: n
      integer r
      integer i
      if(n < 0) then
         write(*,*) 'Received negative input'
         stop
      end if
      r = 1
      do i = 1,n
         r = r*i
      end do
      factorial=r
   end function factorial
end module factorial_mod

Compile:

gfortran -c test.cr.f90

Action: Anne to create a makefile to test the code (and that can be reused for interpolate_sub.f90)

Adding test for interpolate_sub.f90

What kind of tests should we do?

• unrealistic temperatures

Plan:

• implement one simple test
• Add github action to automate testing
• Create and implement more tests

Hackathon day

• Create mybinder environment & badge snippet:

  • go to https://mybinder.org/
  • and fill information as shown below e.g. fill the Github repository URL: https://github.com/annefou/hackathon-south-africa
  

• test mybinder to create test environment

  • Does it work for you?

• Open your web browser and go to https://github.com/annefou/hackathon-south-africa

• mybinder documentation: https://mybinder.readthedocs.io/en/latest/introduction.html

When compiling with mybinder terminal/console, we use $FC environment variable to compile (because the compiler name is weird/difficult to remember /srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-gfortran)

$FC interpolate_sub.f90 -o interpolate.x

Testing/Using pFUnit

There is a very nice repository with examples on how to run/use pFUNIT:
https://github.com/Goddard-Fortran-Ecosystem/pFUnit_demos

Clone the pFUNIT demo repository

git clone https://github.com/Goddard-Fortran-Ecosystem/pFUnit_demos.git

We will now test one example in Trivial folder:

cd pFUnit_demos/Trivial

Edit the Makefile and add -lgomp in my_tests_OTHER_LIBRARIES:

SRCS := square.F90
OBJS := $(SRCS:%.F90=%.o)

all: my_tests

libsut.a: $(OBJS)
	$(AR) -r $@ $?

ifeq (nagfor,$(findstring nagfor,$(FC)))
  FFLAGS += -fpp
endif

%.o : %.F90
	$(FC) -c $(FFLAGS) $<


LATEST_PFUNIT_DIR := $(lastword $(shell echo $(wildcard $(PFUNIT_DIR)/PFUNIT-4.*) | xargs -n1 | sort -V))
include $(LATEST_PFUNIT_DIR)/include/PFUNIT.mk

FFLAGS += $(PFUNIT_EXTRA_FFLAGS)

test_square.o: square.mod
square.mod: square.o

my_tests: libsut.a

my_tests_TESTS := test_square.pf
my_tests_REGISTRY :=
my_tests_OTHER_SOURCES :=
my_tests_OTHER_LIBRARIES := -L. -lsut -lgomp
my_tests_OTHER_INCS :=

$(eval $(call make_pfunit_test,my_tests))


clean:
	$(RM) *.o *.mod *.a *.inc test_square.F90

And then:

First we need to set pFUNIT environment variable:

 export PFUNIT_DIR=$HOME/packages

Then we can compile test and run it:

make clean
make all
./my_tests

It should return:

Time:         0.001 seconds

 OK
 (1 test)

Exercice: make the test fail

For instance, change square.F90:

module Square_mod
contains
   
   pure real function square(x)
      real, intent(in) :: x
      square = x**3
   end function square
end module Square_mod

Then without changing the test, I check if my changes are correct:

make clean
make
./my_tests

And it returns:

 ./my_tests
.F
Time:         0.001 seconds

Failure
 in:
test_square_suite.test_square
  Location:
[test_square.pf:6]
square(3)
AssertEqual failure:
      Expected: <9.00000000>
        Actual: <27.0000000>
    Difference: <18.0000000> (greater than tolerance of 0.00000000)

 FAILURES!!!
Tests run: 1, Failures: 1, Errors: 0
, Disabled: 0
ERROR STOP *** Encountered 1 or more failures/errors during testing. ***

Error termination. Backtrace:
#0  0x55b0ec9d4b03 in __funit_MOD_finalize
        at /home/jovyan/pFUnit/src/funit/FUnit.F90:171
#1  0x55b0ec9e4def in funit_main_
        at /home/jovyan/pFUnit/src/funit/funit_main.F90:17
#2  0x55b0ec98a10f in ???
#3  0x7f66166e2bf6 in ???
#4  0x55b0ec98a160 in ???

The expected value is still 9.0 but now I got 27.0. so the changes I implemented in square.F90 is not correct.

Testing interpolate_sub.F90

1. Which test to add: we will check the subroutine INTERP and check that the values are "correct"

BREAK: 10mn –> 11:45 SA Time

Register your changes with git:

git add interpolate_sub.f90
git add inter_mod.f90

Then we need to setup username and email address for git:

configure git:

git config --global user.name "Your name here"
git config --global user.email "[e-mail]"

Sor for me:

git config --global user.name "Anne Fouilloux"
git config --global user.email "[e-mail]"

Then we "commit":

git commit -m "create a new Fortran module for interp subroutine"

On github: https://github.com/annefou/hackathon-south-africa

we still do not have this change yet so we need to "push" the changes to github:

git push 

Ask you for username (maryjanebopape) and password.

Then it is updated on github.

For everyone else, we need to update our local copy:

git pull

Do you get Mary-Jane's changes on your local "copy"?

And we can now check our local copy works:

Compile:

$FC Inter_mod.f90 interpolate_sub.f90  -o interpolate.x

And run:

./interpolate.x

test for interp

Values at 4 points: 29, 30, 31 and 32
Answer: 30.5
Latitude: south point: -33, north point: -31
Longitude: west point: 20, east point: 22
Station point: 21; -32

Hackathon cont. 30/04/2021

• start software environment with mybinder (takes a few minutes)
• Open a terminal and then clone pFUnit_demos:

git clone https://github.com/Goddard-Fortran-Ecosystem/pFUnit_demos

• modify Inter_mod.f90 and replace subroutine by a function.

 SUBROUTINE INTERP(I,J,K,TEMP,LONS,LATS,LONSTAT,LATSTAT,NEW)
     IMPLICIT NONE
     INTEGER, INTENT(IN) :: I
     INTEGER, INTENT(IN) :: J
     INTEGER, INTENT(IN) :: K
     REAL, INTENT(IN) :: TEMP(I-1:I,J-1:J)
     REAL, INTENT(IN) :: LONS(I-1:I)
     REAL, INTENT(IN) :: LATS(J-1:J)
     REAL, INTENT(IN) :: LONSTAT(K:K)
     REAL, INTENT(IN) :: LATSTAT(K:K)
     REAL, INTENT(OUT) :: NEW(K:K)
!    REAL, INTENT(INOUT) :: LATSSTAT
     REAL :: VAR1
     REAL :: VAR2
       
       VAR1=(LONS(I)-LONSTAT(K))/(LONS(I)-LONS(I-1))*TEMP(I-1,J-1)+&
            (LONSTAT(K)-LONS(I-1))/(LONS(I)-LONS(I-1))*TEMP(I,J-1)
       VAR2=(LONS(I)-LONSTAT(K))/(LONS(I)-LONS(I-1))*TEMP(I-1,J)+&
            (LONSTAT(K)-LONS(I-1))/(LONS(I)-LONS(I-1))*TEMP(I-1,J)
       NEW(K)=(LATS(J)-LATSTAT(K))/(LATS(J)-LATS(J-1))*VAR1+&
            (LATSTAT(K)-LATS(J-1))/(LATS(J)-LATS(J-1))*VAR2

    END SUBROUTINE INTERP

replaced by

 pure real function INTERP(I,J,K,TEMP,LONS,LATS,LONSTAT,LATSTAT)
     IMPLICIT NONE
     INTEGER, INTENT(IN) :: I
     INTEGER, INTENT(IN) :: J
     INTEGER, INTENT(IN) :: K
     REAL, INTENT(IN) :: TEMP(I-1:I,J-1:J)
     REAL, INTENT(IN) :: LONS(I-1:I)
     REAL, INTENT(IN) :: LATS(J-1:J)
     REAL, INTENT(IN) :: LONSTAT(K:K)
     REAL, INTENT(IN) :: LATSTAT(K:K)
!    REAL, INTENT(INOUT) :: LATSSTAT
     REAL :: VAR1
     REAL :: VAR2
       
       VAR1=(LONS(I)-LONSTAT(K))/(LONS(I)-LONS(I-1))*TEMP(I-1,J-1)+&
            (LONSTAT(K)-LONS(I-1))/(LONS(I)-LONS(I-1))*TEMP(I,J-1)
       VAR2=(LONS(I)-LONSTAT(K))/(LONS(I)-LONS(I-1))*TEMP(I-1,J)+&
            (LONSTAT(K)-LONS(I-1))/(LONS(I)-LONS(I-1))*TEMP(I-1,J)
       INTERP=(LATS(J)-LATSTAT(K))/(LATS(J)-LATS(J-1))*VAR1+&
            (LATSTAT(K)-LATS(J-1))/(LATS(J)-LATS(J-1))*VAR2

    END Function INTERP

update test_Inter_mod.pf

@test
subroutine test_Inter_mod()
   use Inter_mod
   use funit
  Implicit None
  Integer:: I=2
  Integer:: J=2
  Integer:: K=1
  REAL :: TEMP(2,2)
  REAL :: LON(2)
  REAL :: LAT(2)
  REAL :: LONSTAT
  REAL :: LATSTAT

  TEMP(1,2)=29.
  TEMP(2,2)=30.
  TEMP(1,1)=31.
  TEMP(2,1)=32.
  LAT(2) = -31.
  LAT(1) = -33.
  LON(2) = 22.
  LON(1) = 20.
  LONSTAT = 21.
  LATSTAT = -32.

  
	!Values at 4 points: 29, 30, 31 and 32
	!Answer: 30.5
	!Latitude: south point: -33, north point: -31
	!Longitude: west point: 20, east point: 22
	!Station point: 21; -32

   @assertEqual(30.5, INTERP(I,J,K,TEMP,LON,LAT,LONSTAT,LATSTAT), 'INTERP(I,J,K,TEMP,LON,LAT,LONSTAT,LATSTAT)')

end subroutine test_Inter_mod

• copy and existing makefile from pFUNIT_demos

SRCS := Inter_mod.F90
OBJS := $(SRCS:%.F90=%.o)

all: my_tests

libsut.a: $(OBJS)
        $(AR) -r $@ $?

ifeq (nagfor,$(findstring nagfor,$(FC)))
  FFLAGS += -fpp
endif

%.o : %.F90
        $(FC) -c $(FFLAGS) $<


LATEST_PFUNIT_DIR := $(lastword $(shell echo $(wildcard $(PFUNIT_DIR)/PFUNIT-4.*) | xargs -n1 | sort -V))
include $(LATEST_PFUNIT_DIR)/include/PFUNIT.mk

FFLAGS += $(PFUNIT_EXTRA_FFLAGS)

test_Inter_mod.o: Inter_mod.mod
Inter_mod.mod: Inter_mod.o

my_tests: libsut.a

my_tests_TESTS := test_Inter_mod.pf
my_tests_REGISTRY :=
my_tests_OTHER_SOURCES :=
my_tests_OTHER_LIBRARIES := -L. -lsut
my_tests_OTHER_INCS :=

$(eval $(call make_pfunit_test,my_tests))

clean:
        $(RM) *.o *.mod *.a *.inc test_Inter_mod.F90

• Try to compile

make

but we got an error because we need to define PFUNIT_DIR

export PFUNIT_DIR=/home/jovyan/packages/

Failed with an error with lonstat because we did not define it as an array of 1. So let's update it:

update test_Inter_mod.pf

@test
subroutine test_Inter_mod()
   use Inter_mod
   use funit
  Implicit None
  Integer:: I=2
  Integer:: J=2
  Integer:: K=1
  REAL :: TEMP(2,2)
  REAL :: LON(2)
  REAL :: LAT(2)
  REAL :: LONSTAT(1)
  REAL :: LATSTAT(1)

  TEMP(1,2)=29.
  TEMP(2,2)=30.
  TEMP(1,1)=31.
  TEMP(2,1)=32.
  LAT(2) = -31.
  LAT(1) = -33.
  LON(2) = 22.
  LON(1) = 20.
  LONSTAT = 21.
  LATSTAT = -32.

  
	!Values at 4 points: 29, 30, 31 and 32
	!Answer: 30.5
	!Latitude: south point: -33, north point: -31
	!Longitude: west point: 20, east point: 22
	!Station point: 21; -32

   @assertEqual(30.5, INTERP(I,J,K,TEMP,LON,LAT,LONSTAT,LATSTAT), 'INTERP(I,J,K,TEMP,LON,LAT,LONSTAT,LATSTAT)')

end subroutine test_Inter_mod

We also need to update makefile:

my_tests_OTHER_LIBRARIES := -L. -lsut -lgomp

but then the test is failing so we need to investigate why:

jovyan@jupyter-annefou-2dhackathon-2dsouth-2dafrica-2dn80iq21v:~$ ./my_tests
.F
Time:         0.000 seconds

Failure
 in:
test_Inter_mod_suite.test_Inter_mod
  Location:
[test_Inter_mod.pf:33]
INTERP(I,J,K,TEMP,LON,LAT,LONSTAT,LATSTAT)
AssertEqual failure:
      Expected: <30.5000000>
        Actual: <30.2500000>
    Difference: <-0.250000000> (greater than tolerance of 0.00000000)

 FAILURES!!!
Tests run: 1, Failures: 1, Errors: 0
, Disabled: 0
ERROR STOP *** Encountered 1 or more failures/errors during testing. ***

Error termination. Backtrace:
#0  0x558a95cfabfc in __funit_MOD_finalize
        at /home/jovyan/pFUnit/src/funit/FUnit.F90:171
#1  0x558a95d0aee8 in funit_main_
        at /home/jovyan/pFUnit/src/funit/funit_main.F90:17
#2  0x558a95cb010f in ???
#3  0x7fa817b28bf6 in ???
#4  0x558a95cb0160 in ???

We chose to test the function itself (it is a bit of a hack e.g. we had to remove pure, etc. but at least we can check the values)

 MODULE INTER_MOD
   IMPLICIT NONE

    CONTAINS

  real function INTERP(I,J,K,TEMP,LONS,LATS,LONSTAT,LATSTAT)
     IMPLICIT NONE
     INTEGER, INTENT(IN) :: I
     INTEGER, INTENT(IN) :: J
     INTEGER, INTENT(IN) :: K
     REAL, INTENT(INOUT) :: TEMP(I-1:I,J-1:J)
     REAL, INTENT(INOUT) :: LONS(I-1:I)
     REAL, INTENT(INOUT) :: LATS(J-1:J)
     REAL, INTENT(INOUT) :: LONSTAT(K:K)
     REAL, INTENT(INOUT) :: LATSTAT(K:K)
!    REAL, INTENT(INOUT) :: LATSSTAT
     REAL :: VAR1
     REAL :: VAR2
       
       
  TEMP(I-1,J)=29.
  TEMP(2,J)=30.
  TEMP(I-1,J-1)=31.
  TEMP(I,J-1)=32.
  LATS(J) = -31.
  LATS(J-1) = -33.
  LONS(I) = 22.
  LONS(I-1) = 20.
  LONSTAT(K) = 21.
  LATSTAT(K) = -32.
       VAR1=(LONS(I)-LONSTAT(K))/(LONS(I)-LONS(I-1))*TEMP(I-1,J-1)+&
            (LONSTAT(K)-LONS(I-1))/(LONS(I)-LONS(I-1))*TEMP(I,J-1)
       VAR2=(LONS(I)-LONSTAT(K))/(LONS(I)-LONS(I-1))*TEMP(I-1,J)+&
            (LONSTAT(K)-LONS(I-1))/(LONS(I)-LONS(I-1))*TEMP(I-1,J)
       INTERP=(LATS(J)-LATSTAT(K))/(LATS(J)-LATS(J-1))*VAR1+&
            (LATSTAT(K)-LATS(J-1))/(LATS(J)-LATS(J-1))*VAR2

    END Function INTERP
    
   END MODULE INTER_MOD

The value returned is still 30.25. Now we see that the interpolation may not be as
simple as we though when we built the test.

We chose to add a tolerance in our test. When interpolation to a station location, we could use different interpolation method but the value should be withi 0.25 C.

!@test
subroutine test_Inter_mod()
   use Inter_mod
   use funit
  Implicit None
  Integer:: I=2
  Integer:: J=2
  Integer:: K=1
  REAL :: TEMP(2,2)
  REAL :: LON(2)
  REAL :: LAT(2)
  REAL :: LONSTAT(1)
  REAL :: LATSTAT(1)

  TEMP(1,2)=29.
  TEMP(2,2)=30.
  TEMP(1,1)=31.
  TEMP(2,1)=32.
  LAT(2) = -31.
  LAT(1) = -33.
  LON(2) = 22.
  LON(1) = 20.
  LONSTAT = 21.
  LATSTAT = -32.

 
	!Values at 4 points: 29, 30, 31 and 32
	!Answer: 30.5
	!Latitude: south point: -33, north point: -31
	!Longitude: west point: 20, east point: 22
	!Station point: 21; -32

#line 33 "/home/jovyan/test_Inter_mod.pf"
  call assertEqual(30.5, INTERP(I,J,K,TEMP,LON,LAT,LONSTAT,LATSTAT), tolerance=0.25, &
 & location=SourceLocation( &
 & 'test_Inter_mod.pf', &
 & 33) )
  if (anyExceptions()) return
#line 34 "/home/jovyan/test_Inter_mod.pf"

end subroutine test_Inter_mod

module Wraptest_Inter_mod
   use FUnit
   implicit none
   private

contains


end module Wraptest_Inter_mod

function test_Inter_mod_suite() result(suite)
   use FUnit
   use Wraptest_Inter_mod
   implicit none
   type (TestSuite) :: suite

   class (Test), allocatable :: t

   external test_Inter_mod


   suite = TestSuite('test_Inter_mod_suite')

   if(allocated(t)) deallocate(t)
   allocate(t, source=TestMethod('test_Inter_mod', test_Inter_mod))
   call suite%addTest(t)


end function test_Inter_mod_suite

Then it worked.

When designing your test:

• Think about the "impact" of the algorithm you choose on the final result. Here, we take 4 values of a variable and interpolate it to a station location. What interpolation algorithm would be valid? What is the tolerance? etc.
• Make sure to comment your functions (add a link to a scientific paper or name the algorithm you use)

Now let's clean Inter_mod.F90

 MODULE INTER_MOD
   IMPLICIT NONE

    CONTAINS

  pure real function INTERP(I,J,K,TEMP,LONS,LATS,LONSTAT,LATSTAT)
     IMPLICIT NONE
     INTEGER, INTENT(IN) :: I
     INTEGER, INTENT(IN) :: J
     INTEGER, INTENT(IN) :: K
     REAL, INTENT(IN) :: TEMP(I-1:I,J-1:J)
     REAL, INTENT(IN) :: LONS(I-1:I)
     REAL, INTENT(IN) :: LATS(J-1:J)
     REAL, INTENT(IN) :: LONSTAT(K:K)
     REAL, INTENT(IN) :: LATSTAT(K:K)
     REAL :: VAR1
     REAL :: VAR2
       
       VAR1=(LONS(I)-LONSTAT(K))/(LONS(I)-LONS(I-1))*TEMP(I-1,J-1)+&
            (LONSTAT(K)-LONS(I-1))/(LONS(I)-LONS(I-1))*TEMP(I,J-1)
       VAR2=(LONS(I)-LONSTAT(K))/(LONS(I)-LONS(I-1))*TEMP(I-1,J)+&
            (LONSTAT(K)-LONS(I-1))/(LONS(I)-LONS(I-1))*TEMP(I-1,J)
       INTERP=(LATS(J)-LATSTAT(K))/(LATS(J)-LATS(J-1))*VAR1+&
            (LATSTAT(K)-LATS(J-1))/(LATS(J)-LATS(J-1))*VAR2

    PRINT*, 'INTERP = ', (LATS(J)-LATSTAT(K))/(LATS(J)-LATS(J-1))*VAR1+&
            (LATSTAT(K)-LATS(J-1))/(LATS(J)-LATS(J-1))*VAR2
    END Function INTERP
    
   END MODULE INTER_MOD