Update to WSL Ubuntu 20.04 "breaks" Fenics connection

With Windows 2004 v19041 comes the vaunted WSL2, so I thought I'd try updating to that to see if performance improved.  I deleted the working WSL1 18.04 Ubuntu that was installed, enabled HyperV (which is required for WSL2), installed Ubuntu 20.04 with WSL2 successfully.  I then installed the Fenics package, again, with no errors or problems.  When I start FEATool and try to run the file that worked previously with Fenics, it tells me in the Fenics load file:

*****************************************************************
File featool-fenics-bY41PUS58K.py created 13-Aug-2020 11:04:57
try:
    from fenics import *
except:
    raise ValueError("Could not find or import fenics.")
*****************************************************************

and when I try to run it anyway, I get:

*****************************************************************
FEniCS solution file(s) not found.
 
Log file output: C:\Users\"username"\AppData\Local\Temp\featool-fenics-bY41PUS58K.log
 
/mnt/c/Users/"username"/AppData/Local/Temp/featool-fenics-bY41PUS58K 13-Aug-2020 11:06:24 : START
 
Traceback (most recent call last):
  File "/mnt/c/Users/"username"/AppData/Local/Temp/featool-fenics-bY41PUS58K.py", line 101, in <module>
    t_start = time.clock()
AttributeError: module 'time' has no attribute 'clock'
 
/mnt/c/Users/"username"/AppData/Local/Temp/featool-fenics-bY41PUS58K 13-Aug-2020 11:06:24 : STOP
*************************************************************************

Any idea what I've done to break it?  I tried going back to WSL1 without success.  I uninstalled Fenics, and reinstalled without success.  Did something change between 18.04 and 20.04 to where the required files are to "import Fenics"?  Everything runs fine with the internal solvers, but it's a shame to lose the Fenics connection.  Any advice you can give would be helpful.
 

Further info:  When I start FEATool  app in Matlab and run the Help -> System Info, it reports that Fenics Version is Not Found.  I installed Ubuntu from the MS Store as directed on their website.  Have they changed where the Ubuntu files are saved between 18.04 and 20.04?  Is there a way to specify a path somewhere to help FEATool find the installation?  I verified that Python3 is installed as well.

A basic search for the error message:

https://duckduckgo.com/?t=ffab&q=AttributeError%3A+module+%27time%27+has+no+attribute+%27clock%27&ia=web

indicates the "time.clock()" function has been deprecated and removed in Python 3.8. So you will probably have to replace these calls or downgrade to python 3.7 or lower.

time.clock is why it won't run and why FEATool doesn't find Fenics?  I didn't make any special Python install, just whatever Fenics installed with itself.  Why doesn't FEATool report that it found Fenics then?

Here's what an m version of the same file produces for an error:

********************************************************
>> tkdepfenics1
Writing file C:\Users\****\AppData\Local\Temp\featool-fenics.xml
Writing file C:\Users\****\AppData\Local\Temp\featool-fenics.py
Deleting FEniCS data and log files.
Error using featool
fenics: Failed FEniCS installation and version check.

Error in featool

Error in featool

Error in fenics (line 60)
[varargout{:}] = featool( 'feval', 'fenics', varargin{:} );

Error in tkdepfenics1 (line 13)
fea = fenics( fea, 'maxnit', 20, 'nlrlx', 1, 'toldef', 1e-06, 'tolchg', 1e-06, 'ischeme', 2, 'tstep', 1, 'tmax', 1000 );
************************************************

Does that shed anymore light on the issue?  What versions do  you recommend for Ubuntu for Windows and  Fenics installs that will work?

Update:  I made sure Python 3.7 was present, and reran the file.  The time issue is resolve, but it still says it can't find Fenics:

********************************************************
ImportError: cannot import name 'sub_forms_by_domain' from 'ufl.form' (/home/*******/.local/lib/python3.8/site-packages/ufl/form.py)
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/mnt/c/Users/*******/AppData/Local/Temp/featool-fenics-Re5m0a1ct6.py", line 7, in <module>
    raise ValueError("Could not find or import fenics.")
ValueError: Could not find or import fenics.
********************************************************

Probably multiple issues here. Time.clock is why your FEniCS script did not run in your Python 3.8+ installation.

As for why FEATool probably does not find your FEniCS installation, the caller string indicates in the FEniCS script dialog box FEATool per default also expects your python interpreter to be available to be started with the "python3" command.

https://www.featool.com/doc/solver.html#fenics_gui

Most likely your new installation does not set this up so you would either have to define an ahell alias, symbolic link, or change the FEniCS call argument to what starts your python3 interpreter.

I'm guessing these are the issues with your new installation. Furthermore it can be that WSL2 compared to WSL1 results in lower performance with respect to disk io while accessing data in Windows drives.

https://github.com/microsoft/WSL/issues/4197

I reverted to WSL1 and changed back to Ubuntu 18.04, so everything is as before except I  now have win 10 v2004 instead of 1909.  Now FEATool finds the Fenics installation (2019.1 per system info), but still won't run - insisting that it can't import Fenics.  When I run the command python -c 'import fenics' from a powershell, it works without error, so I'm at a loss why FEATool can no longer import it even though it sees it now and the python import works.  Any further thoughts?  I'm out of things to try.

*************************************************************
UPDATE:  Got it to run per above.  Had to add the pip package to Ubuntu, the execute the command

pip3 install fenics-ffc --upgrade

to evidently get the fenics python components installed.  Now I'll try it on WSL2 and maybe Ubuntu 20.04. Just following the standard instructions for  installing fenics were not  evidently enough.

*************************************************************
FURTHER UPDATE:  Did some more testing with following results:

 Fenics runs on Ubuntu 18.04 in either WSL1 or WSL2 on Windows 10 v2004.  To get it to run you have to execute the following commands in Ubuntu Bash:

1.  Just after you install 18.04, run sudo apt update and sudo apt dist-upgrade

2.  Install Fenics as follows:
     sudo apt-get install --no-install-recommends software-properties-common
     sudo add-apt-repository ppa:fenics-packages/fenics
     sudo apt-get update
     sudo apt-get install --no-install-recommends fenics
     sudo apt-get dist-upgrade

3.  Go to a Powershell or Terminal window and run python -c 'import fenics'.  If it executes and returns nothing, including no errors, you are good

4.  Back in Bash, run:
     sudo apt install python3-pip
     pip3 install fenics-ffc --upgrade

5.  Repeat (3) to check the Fenics install

Run Matlab then the FEATool app.  Check Help->System Info to see if it sees Fenics 2019.1.  Then run your Fenics solve.  The script window will still show that it didn't import Fenics, but it did.  When you execute Solve in Auto mode, the execution window will open and repeat a line several times that it is calling FFC and running the JIT compiler (it only seems to do this the first time you run after installing Fenics in Ubuntu), then it will solve the file.

I can't get this to run on Ubuntu 20.04.  As was pointed out, Python 3.8 has dropped the clock function, and errors out in FEATool when it's called for.  In addition, try as I might, I can't get FEATool to locate Fenics from 20.04.  It says it already has the fenics-ffc installed, but it doesn't run it.  My limits on Ubuntu knowledge prevent me from taking that any further.  For now, 18.04 works.  Hope this helps others.

Thank you for sharing your steps to get FEniCS to work again. I'm gussing the the issue with Ubuntu 20.04 WSL might be that "bash.exe" points to your 18.04 WSL. I've summarised the steps that must be working from the point of FEATool below:

FEATool checks for a valid FEniCS installation with the following Matlab system command:

    [st,res] = system('bash -c "dolfin-version"');

The actual FEniCS Python scripts are essentially run with:

    [st,res] = system('bash -c "python3 ''path_to_featool-fenics-script.py''"');   # Windows

    [st,res] = system('python3 ''path_to_featool-fenics-script.py''');             # Linux/Mac OSX

For everything to work smoothly the following must be working.

Windows:

(1) Windows 10 system with "Windows Subsystem for Linux (WSL)" installed.
(2) Working Python 3 and FEniCS installations in the chosen WSL.
(3) The command "python3" aliased or linked to the Python installation with FEniCS (in the chosen WSL).
(4) The "bash.exe" command (C:\Windows\System32\bash.exe) must also point to the WSL installation with the valid FEniCS installation (if several WSL distributions are present).

Linux/Mac OSX:

(1) Working Python 3 and FEniCS installations.
(2) The command "python3" aliased or linked to the Python installation with FEniCS.

If everything is set up correctly the following Matlab command should run without error:

    assert( system('bash -c "dolfin-version"')==0 & system('bash -c "python3 -c ''from fenics import *''"')==0 )

When using Python 3.8 or later the two "time.clock()" commands in the Python script must also be changed to "time.process_time()".

Again, I suspect that WSL2 might be slower due to the issue with Windows/Linux reduced disk io performance in WSL2 I mentioned above (The only way to avoid that until Microsoft can fix it would be to run everything in WSL, Matlab, FEATool, FEniCS etc.).

Thanks for that detailed feedback.  My comments:

- Regarding 20.04 point to 18.04, I did not have 18.04 and 20.04 installed at the same time.  When I tested 20.04, I had uninstalled 18.04 and checked with wsl -l -v that bash didn't see any linux installation.  When I did try to have both 18.04 and 20.04 simultaneously installed, there were some unpleasant interactions, so I stopped doing that.  For instance, when I had installed fenics in 20.04 and went to access ppa in 18.04, it wouldn't do it - it just timed out waiting for a "key".

- Fenics showed proper installation into the WSL for each version using the python -c 'import fenics' command from the powershell window.

- 20.04 never showed fenics installed using the FEATool system info command, but 18.04 did.

- 20.04 showed that the fenics -ffc was already installed but wouldn't run when called, where 18.04 didn't have it until I installed it, then ran it OK

- I saw no speed difference between wsl and wsl2 for my problems.  The comments in the link you quoted were pretty old, and most trying wsl2 then were "windows insiders", not using the official release of wsl2.  Maybe the issue has been fixed.  I don't usually follow that stuff unless I need to investigate something like this.

- Right now, I'd prefer to run 18.04 without having to edit the script each time for that time command, unless there is a way to do the edit once to the script creator.

MS still says wsl is in flux, as they are still working to allow a gui interface (only command line for now), so I anticipate more changes.  For now, I'm happy to have a working interface, albeit in 18.04.

Have you seen it work in 20.04 in either wsl or wsl2 using the commands you list?

tkdiff wrote

- 20.04 never showed fenics installed using the FEATool system info command, but 18.04 did.

That is odd if you only had one WSL installation as it should indicate that the windows "bash" command pointed somewhere else.

tkdiff wrote

- Right now, I'd prefer to run 18.04 without having to edit the script each time for that time command, unless there is a way to do the edit once to the script creator.

I will fix the timing deprecation in the next update.

tkdiff wrote

Have you seen it work in 20.04 in either wsl or wsl2 using the commands you list?

I didn't try WSL2 yet, but run Linux development and testing on WSL1 (Ubuntu 18.4 and Alpine Linuxes). From my understanding WSL2 is faster if you only work in the Linux "partition" (like a VM which it essentially is), while WSL1 is still better for mixed Windows/Linux workflows which I do (and WSL2 does not work with VirtualBox VMs either, since it requires Hypervisor enabled). One performance tip though is to disable Windows defender scanning from the WSL linux system folders as it is kind of unnecessary).

tkdiff wrote

MS still says wsl is in flux, as they are still working to allow a gui interface (only command line for now), so I anticipate more changes.  For now, I'm happy to have a working interface, albeit in 18.04.

Although maybe not officially supported running GUI apps has been working almost since the beginning of WSL1 if you install and start and X server such as VcXsrv or Xming, before starting the WSL:

https://sourceforge.net/projects/vcxsrv

The Fenics connection broke again after a doing a routine update to Ubuntu 18.04 with the following commands:
sudo apt update
sudo apt upgrade

When these two completed, some messages were present with something about a dolfin2019 lib was no longer needed and removed.  Fenics still shows installed properly.

When I run your your test assert command in matlab:

assert( system('bash -c "dolfin-version"')==0 & system('bash -c "python3 -c ''from fenics import *''"')==0 )

I get:  

2019.2.0.dev0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/fenics/__init__.py", line 7, in <module>
    from dolfin import *
  File "/usr/lib/petsc/lib/python3/dist-packages/dolfin/__init__.py", line 144, in <module>
    from .fem.assembling import (assemble, assemble_system, assemble_multimesh, assemble_mixed,
  File "/usr/lib/petsc/lib/python3/dist-packages/dolfin/fem/assembling.py", line 38, in <module>
    from ufl.form import sub_forms_by_domain
ImportError: cannot import name 'sub_forms_by_domain'
Assertion failed.

Fenics was running fine just up until this Ubuntu update.  Any thoughts/suggestions?

UPDATE:  I found the following in a google search, where someone else had the same issue:

I have followed where does Python import the ufl package from and I have found another version of it located here:

/home/username/.local/lib/python3.6/site-packages

I have removed it and it seems to be working now.

I'm not an Ubuntu expert - how do I go in and remove this unwanted "ufl package"?

Thank you for reporting the issue and potential workaround. I am testing FEniCS on Ubuntu 18.04 (WSL1) and so far did not observe this issue (but that particular update might not have triggered yet). I'm not sure how to officially remove such an offending package (as manually removing Python packages is potentially error prone, and I think it should have been handled by the updater). Now that it is working though it is maybe safer to not "distrub" the installation at the moment, but otherwise it might work better if one first uninstalls fenics and related packages before re-installing it again, with something like

  sudo apt-get -y purge fenics 
  sudo apt-get -y autoremove 
  sudo apt-get update

Sorry if I was confusing, but I did not attempt the workaround - it's still not working for me.  The line "I have removed it" was from the OP in what I found.  I am reluctant for the reasons you mention to mess with python packages.  I will try your suggestion to remove and reinstall fenics, and see if that fixes it.  It's not working yet, so I can't make it worse.  BTW, I am running Ubuntu  18.04 WSL2 - don't know if that matters.  Like I said, everything was fine until the Ubuntu update.

I see, I don't think WSL1/2 would make a difference as the WSL just cleverly translates system calls to Windows equivalents and therefore should not really see any difference.

You should be able to list all installed Python packages with the command pip list/pip3 list and see if you have anything strange (like multiple copies of ufl). I get the following fenics related packages

  fenics-dijitso (2019.1.0)
  fenics-dolfin (2019.1.0)
  fenics-ffc (2019.1.0.post0)
  fenics-fiat (2019.1.0)
  fenics-ufl (2019.1.0)

Maybe you have already, but I guess it could also be good to ask fenics developers/forums as I guess they are the ones who are actually building and packaging the distribution and should perhaps know better what might be going on.

I get the same list of files....except for dolfin I get fenics-dolfin (2019.2.0.dev0).  I wonder if that's the  difference?  I installed from the ppa repository as directed on the fenics website.

UPDATE:  Uninstall and reinstall has no effect - still show fenics-dolfin (2009.2.0.dev0), and that is the first thing reported in your assert query.  Could it be that you just need to change the version check?

Possibly they have changed something significant in the fenics api in the 2019.2.0.dev0 update. Can you confirm that fenics models exported from FEATool run fine on its own (export the ".py" and ".xml" mesh files, and run them in WSL "python featool-fenics.py"? Otherwise just changing the test/check will not help and it could be and issue with fenics or the update build.

I exported the two files to a windows directory, but I'm not sure how to run them in the ubuntu bash terminal.  Can you explain?

update:  Check out this link....someone had a similar issue with the dolfin update:

https://fenicsproject.discourse.group/t/dolfin-version-2019-2-0-dev0/4308/4

The FEATool-FEniCS solver interface "just" translates the PDE syntax to equivalent FEniCS Python scripts (along with Dolfin xml mesh files), then makes system calls to Python to run the scripts, monitor, and import the solution when done. So you can run the same scripts manually without using the FEATool GUI/CLI interface. If you for example have exported the fenics simulation script and mesh to C:\temp\test.py / .xml, you should be able to run the script by calling "python /mnt/c/temp/test.py" from the WSL bash command prompt (or python3 depending on which Python interpreter alias fenics is installed in).

For what its worth, I have forced an update to fenics 2019.2.0.dev0 and so far have not been able to observe any issues with respects to calling the fenics solver from FEATool.

I ran the python command on the exported file, and it returned:
Traceback (most recent call last):
  File "/mnt/d/docs/test1.py", line 7, in <module>
    raise ValueError("Could not find or import fenics.")
ValueError: Could not find or import fenics.

I can't explain why your setup runs the new dolfin OK and mine does not - I haven't  done anything special other than run sudo apt update and sudo apt upgrade as I usually do every now and then.  Others are evidently having the same issue running fenics in ubuntu 18.04 with the same fail, so it's not an uncommon problem.  Not sure what else I can do now.

UPDATE:
I executed the suggestions to remove ufl in this post:
https://fenicsproject.discourse.group/t/ubuntu-18-04-importerror-cannot-import-name-sub-forms-by-domain/4257/6

When I removed ufl, still nothing worked.  Then I removed and reinstalled fenics, and the following happened:
1. sysinfo  from featool app still shows that it can't find the fenics installation
2. The file using fenics solver now runs from within featool
3. The corresponding m file (exported from featool to matlab) does not run - says can't find fenics
4. Running the exported files from (2) still doesn't run, indicating can't find fenics

In summary, I can now solve with fenics from featool, but nowhere else

Out of curiosity, when you run pip3 list, what does distro-info show for you?  I have 0.18ubuntu0.18.04.1

That error seems to indicate you tried to run the script in a python environment where fenics isn't installed (confusingly a system can have many diffrent versions of python). It should be consistent with that if your "pip list" indicates fenics modules then it is "python fenics-script.py" but if it is "pip3 list" then you should use "python3" instead.

If many fenics users are having the same issue, then probably the issue will eventually surface and a update/fix will be issued by them. Reporting your issue might also help as it seems it has not much to do with the FEATool-FEniCS integration.