Quantcast
Channel: The NAG Blog
Viewing all 81 articles
Browse latest View live

A cure for the common code: NAG at The Trading Show Chicago

$
0
0
A few weeks ago, my colleagues and I attended The Trading Show Chicago, which looks at the world of quantitative financial analysis, trading strategy and technology.  The conference was preceded by two technology workshops; I attended one of these, which was on the use of independent component analysis (ICA) and so-called second order statistics for hedge fund data.  After the session, I had an interesting conversation with the presenters (Prof Andrew Kumiega from IIT and Dr Greg Sterijevski from UBS O’Connor) afterwards about ICA, NAG’s functionality – with which they were already familiar, and complementary about – and the way in which it could be used in ICA.

John Morrissey, Paul Hipes and Brian Spector offering the benefit of their numerical expertise in the NAG Clinic at  The Trading Show Chicago

Following a plenary session on the opening day of the conference, the proceedings were split into four streams: Trading Congress, HFT World, Quant Invest and Exchange Technology.  There was a very nice talk by Professor Emanuel Derman on the use of metaphor, models and theory by quants; I had a brief chat with Derman afterwards, during which he referred to NAG in positive terms.  After the conference, my colleague kindly lent me Derman’s book, "My Life As A Quant" which provides some valuable insights on his evolving views about the applicability of models to quantitative analysis, as well as interesting background on his personal history (he started his academic career as a particle physicist, switching to finance when he joined Goldman Sachs in 1985 to become one of the first so-called POWS – i.e. Physicists On Wall Street).

Another good talk was that by Ram Ahluwalia on human bias and machine learning for hedge funds, which contained some stimulating technical detail on random matrix theory and principal components analysis.  His blog is built around questions and answers for finance professionals and academics; for example, a recent query about finding the nearest correlation matrix contained an approving reference to a paper by Professor Nick Higham and his colleagues on the subject (the method described therein has just been implemented in the latest release of the NAG Library - see, for example, here).

Brian Spector explaining an abstruse technical point to me in the NAG Clinic.

My seminar – “A cure for the common code”, which described NAG’s offerings for quants, and suggested that using some of our routines to provide the 'numerical heavy lifting' would allow developers to concentrate their effort on the more domain-specific parts of their application – was well-attended, and seemed to go over well, judging by the number of questions asked.  Following the talk, many attendees availed themselves of the opportunity to visit the nearby NAG booth.

This meeting was the first occasion on which the booth was supplemented by the NAG Clinic, which offered a complementary consulting resource that offered advice and insights on business problems requiring sophisticated numerical analytics and computation.  The clinic was staffed by John Morrissey and Brian Spector from the technical team at NAG Inc., Paul Hipes, who's an independent consultant providing specialist analytical software (often using NAG's components) to the financial services industry, and (intermittently) myself.  The discussions with attendees yielded valuable insights into the detailed numerical requirements of quants, and gave us many useful further contacts in the field.

Finally, as in the case of the INFORMS meeting earlier this year, I found that the conference location (on Chicago's Navy Pier, just along from the Ferris Wheel, the Smith Museum of Stained Glass, and next to the beer garden) provided almost as much stimulation as the meeting itself.

What's new at Mark 23? Linear quantile regression

$
0
0
This is the first in an occasional series of posts highlighting new functionality in the latest release (Mark 23) of the NAG Library.  Here, we describe linear quantile regression, which has just been added to the collection of regression techniques that is already available in the Library.

Regression techniques are concerned with modelling and analyzing the relationship between a dependent (or response) variable and one or more independent (or explanatory) variables.  More specifically, they enable the user to understand how the typical value of the response variable changes when one or more of the explanatory variables are varied.  A common example of regression analysis is linear least-squares regression, which is concerned with modelling the behaviour of the conditional mean of the response variable and which, as its name implies, employs the well-known method of least-squares.  By contrast, linear quantile regression models one of the conditional quantiles of the response variable (see this note for more information on its formal definition).  The solution of this problem is obtained using linear programming techniques; least-squares methods cannot be used here.

Because quantile regression allows multiple quantiles to be modelled, it can provide an analysis of the relationship between variables which is more comprehensive than that given by least-squares regression, since that only considers the behaviour of the mean.  For example, the results of least-squares regression will not be as robust as those of quantile regression if the conditional distribution of the response variable has heavy tails, is asymmetric, or is not unimodal.  Quantile regression is less sensitive than least-squares regression to outliers in the response variable and, by concentrating on specific quantiles, the user is able to investigate the behaviour of different parts of the distribution.  Applications of quantile regression include its use in finance to estimate so-called Value at Risk (see, for example, here, here and here), and it has been used in ecology to discover predictive relationships between variables in cases where the relationship between their means is either non-existent or only weak.

In Mark 23 of the NAG Fortran Library, quantile regression is performed by the G02QGF routine and by G02QFF, which provides a simplified interface to G02QGF.  (The corresponding routines in the latest versions of the NAG C Library are nag_regsn_quant_linear and nag_regsn_quant_linear_iid, and of the NAG Toolbox for MATLAB are nag_correg_quantile_linreg and nag_correg_quantile_linreg_easy).  The figure below shows the results from the example for these routines, which investigates the relationship between household food expenditure and income, using data from an 1857 study.

Plot of household food expenditure versus income, along with the results of fitting a quantile regression model to the data for the five quantiles indicated in the key.

I'm grateful to my colleagues Martyn Byng and Jeremy Walton for their help in preparing this post, and their interest in this work.

NAG on the Cloud: Part 2

$
0
0
In case you missed Part 1 of NAG on the Cloud, I discussed calling the Library on Windows Azure using the C# library and Microsoft's Cloud Numerics. We now turn to a different cloud provider. Amazon's Elastic Cloud Computing (EC2) allows the user to buy or rent instances of virtual computers and pay for only what you use. Below, I'll discuss how to put the NAG Library on EC2 and give some initial performance tests.

Creating an Instance 
There are a plethora of videos on creating Amazon EC2 instances. If you are going to install and run the NAG Library, make sure you start a Virtual Private Cloud when creating the instance. Private Clouds can be created at no additional cost on EC2 and remain compatible with NAG's Kusari License management system. I decided to create a couple tests:

My EC2 Console Instances

The above instances vary in size, memory, and type of OS running. When the instance is created, you can gain access to it via ssh. To load the library onto EC2, just download the .tgz file of your choice from the NAG website and secure copy it up there:

$scp -i VPCkey.pem fll6a23dfl.tgz ubuntu@50.112.131.89:/home/ubuntu

Here VPCkey.pem is a key file that only allows my computer to access this Cloud instance. Once the Library is copied, we can unzip it and the installation instructions remain the same as if the computer were right in front of you!

Is the Cloud Faster?
First the system specs:
Amazon EC2 Cloud High-CPU Extra Large Instance:
  • 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 compute units each)
  • 64-bit Ubuntu
  • 7 GB of memory
My laptop:
  • Intel Core i5
  • 64-bit Ubuntu
  • 3 GB RAM 
Clearly the Cloud is more powerful, but is it faster? Since I had recently completed a project on sparse Linear and Quadratic Programming, I decided to use some of the data and compare the run-times between my laptop and the cloud (routine tested was e04nq, all times are in seconds):

Cloud Time My Laptop

QP1 1114.42 1429.06

QP2 45.97 57.2

LP1 78.13 96.83

LP2 24.36 28.99

LP3 39.42 43.53
Thus, by running these programs on the cloud I was able to obtain a 10-20% speed improvement!

CPU Utilization on the Cloud
As I was browsing the information page of my High-CPU Instance, I examined the CPU utilization used in the above programming problems and found some interesting results (click to enlarge):


As you can see, I was using less than 20% of my CPU capacity on the Cloud when running these tests! This is a result of using the serial library and not taking advantage of all the cores on EC2. I would suspect you could obtain a substantial performance increase when using the NAG Library for SMP and Multicore.

For more information on running your applications on the Cloud, contact support@nag.co.uk or support@nag.com.

The NAG SMP and Multicore Library on the Cloud

$
0
0
In my last post we looked at the NAG serial library on Amazon's EC2. We noticed that the CPU utilization was not anywhere close to the capacity I had on the Cloud. Below, I have loaded the NAG SMP and Multicore Library on Amazon's EC2 in hopes of utilizing all the virtual cores.
Routines Tested
The EC2 instance used was a High-CPU, c1.xlarge instance (7GB of memory and 8 Virtual Cores). The SMP Library contains 204 tuned and a total of 337 enhanced routines for use on Multicore machines from which I tested f11me(Sparse Matrix Factorization) and g02bn(Kendall/Spearman rank coefficient). While I had a maximum of 8 cores available, I decided to increase the number of threads beyond this, just to see the result. Below you will see how the time taken scales as the number of threads increases (click to enlarge):


Both routines scale well as you increase the number of threads, but f11me takes longer with 12 threads as opposed to 8! I suspect the slowdown is a result of dependencies between threads. In order for some to start, they may have to wait on other parts of the matrix factorization to finish. G02bn on the other hand doesn't require communication between threads so each one can run independently. This routine slightly benefited from running on 12 vs. 8 threads.



This raises an important point: One should always approach parallel programming cautiously. Some programs can take longer across a number of threads and even return an incorrect answer if the routine is not thread safe (thankfully, the NAG SMP Library is thread safe so we don't have to worry).

 CPU Utilization
In my last post we saw the CPU utilization didn't exceed 20% of the maximum. This left me slightly uneasy, as I was paying for 8 cores, but I could not use all of them! The CPU utilization for the above test looks better:


Ahh finally, we are able to use all the virtual cores on EC2.

One last interesting note is that during the initial testing of the Library on the Cloud, I ran into an allocation error with the 600 MB of memory available (my initial testing always uses the free tier before scaling up to a High-CPU instance). The allocation error can be easily remedied via changing the instance type. It takes only minutes to scale up to a large, cluster compute, or high memory instance.

In Summary
-The NAG SMP Library works on EC2 and runtime scales well across the number of cores.
-Amazon's EC2 is a powerful cloud computing tool that allows you to easily increase Memory/CPU capacity (obtaining a maximum of 70GB memory or 16 cores). 
-Care should be exercised when parallel programming. 

The making of “1000x” – unbalanced supercomputing

$
0
0
I had a bit of a rant in my article published at HPCwire this week - “Chasing1000x: The future of supercomputing is unbalanced”.

The gist of my rant is that the supercomputing community pays great attention to the next factor of 1000x in performance – and I firmly agree that next 1000x is highly valuable to the HPC community and the wider economy. But, we should give equal attention to 1000x in other areas, notably ease-of-use and growth of the user-base. And, critically, give equal peer recognition to those promoting that growth and pursuing ease-of-use and ease-of-access, not reserve all our “visionary” accolades for those figuring out the details of the first exascale computers.

However, I planted an appropriate pun in the title of the article itself. The obvious meaning, in the context of the article, is that the future of supercomputing is unbalanced with respect to the focus on performance versus community growth etc. However, the double meaning should be readily recognizable to anyone active watching or developing the HPC technology roadmaps. The future of supercomputers is unbalanced – i.e., the broad summary of the many technology roadmaps out there is that future supercomputers (at all scales) will be less balanced in some key performance attributes than current technology.


There may be other options but, based on the current evidence of technology development and indications of acceptable power and costs constraints, the most likely future is as follows.

The peak calculating capacity of processors will keep growing exponentially – an apparition of the industry’s old friend Moore’s Law. (For most scientific/engineering use, this means floating point operations per second - or FLOPS.) The energy efficiency of this peak capacity will also dramatically improve - FLOPS/watt. However, while the performance and capacity of the data side of the system will continue to grow - they will not grow anywhere near as fast as the availability of FLOPS.

Thus, we can assume that future HPC systems - from desktop workstations to the fastest supercomputers - will be abundant in FLOPS but moving data around the system will incur significant (relative) time and energy costs. Getting some data from memory (e.g. to do a calculation with) will cost much more time and energy than doing the calculation itself. The performance penalty (due to latency and bandwidth limits relative to the increase processor speeds) will mean that computers will appear significantly unbalanced compared to current systems.

For more background on the architectural changes, there are many presentations talks worth looking at, including for example several by Jack Dongarra or this by David Keyes.

The key message is that software implementations (including algorithms) required to get the best performance out of these future architectures are likely to be different to those in the bulk of current applications. This is because those current implementations make certain assumptions about the relative costs of FLOPS, bandwidth, latency, etc. (based on today's hardware) - and those assumptions will no longer be valid.

This imbalance will probably affect the performance of applications - at the worst, without modification, current software implementations could run slower on future hardware! I am not suggesting this extreme will be a common occurrence. But it is very likely that, without effort, most current application software will fail to take full advantage of the performance potential of future HPC systems. Quite simply, due to the data movement challenges, they will not be able to use all of those FLOPS thrown at us by Moore's Law.

Thus, to continue achieving major performance improvements ("1000x"), the users of modeling and simulation, the developers of software applications, and the providers of HPC services and systems will all have to investigate the role of software innovation as an integral partner with the hardware speed race.

And the same applies to the core theme of my article in HPCwire (growth in the users of HPC and improvements in the ease-of-use will deliver as much or more to the economic good than chasing the high end performance alone). Just like 1000x in performance, those goals of 1000x users and 1000x ease-of-use also require innovation in the software to be an integral part of the approach.

Now the games are over – what’s our legacy? How can we inspire a generation?

$
0
0
London 2012 Olympic Park
I really didn’t expect to love the Olympics as much as I did. I’m not really into sport and have never taken much interest before, but this time I was absolutely enthralled. London, being the host city, obviously had a bearing on my enthusiasm and enjoyment, but what now the games are over? A lot of talk centred around the Olympics 2012 ethos of ‘inspiring a generation’ and this got me thinking about NAG’s legacy and how we too can inspire.

NAG has had a busy year as well; we've managed to produce significant new releases of our four main numerical software products, the C and Fortran Libraries, the Library for SMP & Multicore and the Toolbox for MATLAB®. If you study the history of the Library you start to understand how truly amazing it is. So is the Library NAG’s legacy? Well, funnily enough, our legacy is right there in each of the new releases. That’s why the NAG Library is amazing. It’s an ever changing, ever improving set of codes that have been developed by people that work here or contributed by well-known and maybe not so quite well-known numerical analysts, statisticians and computer scientists from all around the world.

Back in the day – 1970 to be precise – Brian Ford, NAG’s Founding Director, was inspired with a vision of a collective inter-university numerical library and set about creating the first NAG Library. Mentored by the legendary Jim Wilkinson, Brian established NAG as an organisation that was, and still is not-for-profit. Brian said of Jim,
“he gave us his invaluable numerical linear algebra software, and his contacts".
Brian's key ideals were voluntary collaboration and quality in every phase of the activity. Over the years mathematical and statistical routines have been contributed by some highly regarded people:

Optimization contributions, enabled by David Martin, came from Walter Murray and Phil Gill of the UK National Physical Laboratory (NPL). Further contributions from NPL came from Geoff Hayes and Maurice Cox in Curve and Surface Fitting. Ian Gladwell contributed in the area of differential equations and Brian Smith in polynomial root finding. We also gratefully acknowledge the considerable work of Elise De Donker for quadrature, Nick Maclaren for random numbers, Lawrie Schonfelder for special functions and NAG’s own Sven Hammarling and Jeremy Du Croz, along with Jack Dongarra and Jim Demmel, for their work on linear algebra and LAPACK (these are only a few of the many, many great NAG code donators).

Fast forward to the current day and code contribution remains a cornerstone of our activities. At the recent AGM I watched a fantastic presentation on ‘Algorithmic Differentiation’ by Professor Uwe Naumann of the University of Aachen, who collaborates with NAG on bespoke versions of the Library. There’s also the significant work of Professor Nick Higham in the new nearest correlation matrix and functions of matrices available in the latest marks of the Libraries.

So how do we 'inspire a generation' in 2012? Well, we have a vision to elevate co-operative working to a new level and promote code contribution. We want to see new people donate their codes for evaluation and possible inclusion in the NAG Library, so if you’re reading this and feel inspired, get involved. Email me and we can talk about the ways of working together, or sign up to our email newsletter for future announcements.

Bitwise Reproducibility with the NAG Libraries

$
0
0
I've written in this blog before about the problems of Wandering Precision - where the results computed by a program are not consistent, even when running exactly the same program on the same machine with the same data several times in a row.

At SC12 in Salt Lake City a couple of weeks ago I took part in a "Birds of a Feather" session organised by Intel, where problems like these, associated with bitwise reproducibility of results, were discussed. On the panel, apart from myself, were representatives of Intel's Math Kernel Library, The MathWorks (producers of MATLAB), German engineering company GNS, and the University Of California, Berkeley.

We all gave brief presentations discussing the impact of non-reproducible results on our work or on our users, and ways around the problem. It turned out that most of
our presentations were remarkably similar, involving tales of disgruntled users who were not happy about accepting such varying results. This is true even when the same
users completely understand the fact that numerical software is subject to rounding errors, and that the varying results they see are not incorrect - they may be equally good approximations to an ideal mathematical result. But they just don't like inconsistency - in fact, often they would prefer to get an answer that is accurate to fewer digits of precision so long as it is consistent, rather than inconsistent but slightly more accurate results.

As it happens, we do have some control over these inconsistent results, which are largely due to optimizations introduced by clever compilers that are designed to take advantage of fast SIMD instructions like SSE and AVX that are available on modern hardware. By using appropriate compiler flags on the Intel Fortran and C compilers, for example, we can avoid these optimizations, at the cost of making the code run up to 12 or 15% slower (according to Intel).

For NAG Libraries, we've decided what we're going to do in future. Most NAG Library products are distributed in two variants - one which is based on fast vendor library kernels (like MKL) and one that is not, but consists of all-NAG versions of routines like BLAS and LAPACK. Typically we expect the all-NAG variant to run slower than the MKL-based variant, so what we plan to do for the next Marks of our libraries is to compile the all-NAG variant library avoiding the SIMD optimizations, but compile the MKL_based library still to use them. That way, we hope to get the best of both worlds - our users can choose whether they want better consistency, or better performance.

The first NAG Library product to be affected by this decision will be Mark 24 of the NAG Fortran Library, which is scheduled for release to users sometime in the first part of 2013.

Our own Hall of Fame – the NAG Life Service Recognition Award

$
0
0
A bit of background…

NAG was established as a not-for-profit organisation and remains so, which means it has no shareholders or investors to answer to. All surpluses made are ploughed back into R&D. NAG is ‘owned’ by its members of which are people that founded NAG, current and past employees, collaborators and friends of the company. This all serves to make NAG a pretty unique place to work which might be a reason for the longevity of serving staff (NAG also happens to be an Investors in People Gold company).

It’s certainly not rare for those of us working here to see our colleagues reach the milestone that could be considered ‘lifetime service’. Many staff work at NAG for the majority of their careers and continue to do so well into their ‘golden years’.

Rewarding greatness…

During NAG’s 40th anniversary celebrations in 2011, a NAG ‘Hall of Fame’ was established to recognise outstanding efforts and commitment to the company by a member or collaborator of NAG. The Life Service Recognition Award has now been bestowed twice.



The recipients of the Life Service Recognition Award are selected by a committee, which considers the merits of nominations invited annually from the entire membership. Founder Members of the award are Dr Brian Ford, Founding Director and former Chairmen, Dr David Hartley, Dr Richard Field, and Professor Joan Walsh.

The first recipient (2011) was NAG’s very first employee, Dr Steve Hague.

Steve was NAG's first full-time employee, joining NAG in 1971. Over the years Steve held a number of technical, administrative and managerial roles, as NAG developed into a world leader in its field. These roles included responsibility for a variety of company activities, including many internal projects and various external collaborative activities, e.g. leading international software projects in the USA and Europe. His main technical interests and contributions were in the area of numerical software engineering, portability in particular, and this work was recognized in the award of a D.Phil. by Oxford University in 1982. Steve formally retired from the position of Chief Operating Officer of NAG Ltd in June 2007.

The 2012 award was bestowed upon Mr Jeremy Du Croz.

Jeremy joined NAG in 1975, and worked on most areas of the NAG Library. He is known for his work on linear algebra in the form of fundamental contributions to the BLAS and LAPACK packages. He made considerable impact on the NAG Library in the areas of Fast Fourier Transforms, data fitting and random number generation. Jeremy was a very much respected manager of the Numerical Libraries Division and is remembered for his wisdom and attention to detail. It is a mark of his influence that, for years after ill-health forced him to retire, people asked, and continue to ask, in all kinds of situations, "What would Jeremy have done?"

A place for you?

If you like the sound of NAG and are interested in working here and maybe one day entering the NAG Hall of Fame yourself, check out the Careers at NAG area of the website; if there aren’t any current vacancies, why not email your CV anyway?












Matrix Functions in Parallel

$
0
0

Last year I wrote a blog post about NAG’s work on parallelising the computation of the matrix square root. More recently, as part of our Matrix Functions Knowledge Transfer Partnership with the University of Manchester, we’ve been investigating parallel implementations of the Schur-Parlett algorithm [1].
Most algorithms for computing functions of matrices are tailored for a specific function, such as the matrix exponential or the matrix square root. The Schur-Parlett algorithm is much more general; it will work for any “well behaved” function (this general term can be given a more mathematically precise meaning). For a function such as
using the Schur-Parlett algorithm should be quicker than computing the individual components separately.  This algorithm made its first appearance in Mark 23 of the NAG C Library. Our parallel improvements will be found in Mark 24 of the C Library, next year.
At the heart of the algorithm lies a Taylor series (which is truncated when it has converged sufficiently). For example:
Before this however, the matrix A is converted into upper triangular Schur form. This serves two purposes. It reduces the number of arithmetic operations required to evaluate the Taylor series. It also gives us the eigenvalues of A. The Taylor series will converge fastest when the eigenvalues lie close together. The clever bit of the algorithm is to rearrange the Schur form of A, grouping similar eigenvalues together. The diagonal part of the matrix is then split into multiple different sized sub-matrices according to this grouping. A Taylor series is evaluated for each of these diagonal blocks. The off-diagonal blocks are then computed by solving the ‘Parlett recurrence’. This is all very confusing and is best explained by the diagram below:
Here we can see the original matrix (in black), the triangular Schur form (in red), the blocking strategy (in blue) and finally the computation of the off-diagonal blocks (in green). The exact distribution of eigenvalues and therefore the number and size of diagonal blocks is totally dependent on the input matrix.

 
So, what does this mean for parallelism? Clearly, all BLAS calls e.g. matrix multiplications can be done in parallel. But on a higher level, the diagonal blocks can be computed in parallel and then the Parlett recurrence can also be solved, a block superdiagonal at a time, in parallel.
To investigate this higher level parallelism, I constructed various upper triangular 1000 by 1000 test matrices, choosing their eigenvalue distributions carefully. Here is a very brief selection of results from a machine with 8 available threads:
  • 1 block of size 1000: no speed up available
  • 20 blocks of size 50: 5x speed up on 8 threads compared to 1 thread
  • 1000 blocks of size 1: 5x speed up on 8 threads compared to 1 thread
These results are not surprising. For a matrix with only one diagonal block (all eigenvalues close together), nothing can be done in parallel. For matrices with more separated eigenvalues (and therefore multiple diagonal blocks) the evaluation of the diagonal blocks and subsequently the Parlett recurrence (which typically accounts for well over 90% of the run time) and can be done in parallel.
In the future, nested parallelism is likely to become more and more important, and this is something which still needs investigating. Would there come a point whereby it is useful to turn off the threaded BLAS and divert all resources to higher level parallelism, and vice versa? We don’t know yet.
I concluded my last post with the message that the best algorithm for serial architectures may not be the best algorithm for parallel architectures. I think the general point to take from this post is that the performance gain you get from parallelism can be highly dependent on your input data.
[1] P. I. Davies and N. Higham. A Schur-Parlett algorithm for computing matrix functions. SIAM J. Matrix Anal. Appl., 25:464-485, September 2003.

NAG Routines in Different Precisions

$
0
0
Written by David Sayers, NAG Principal Technical Consultant


How would you like to call a NAG routine and have the precision it computes in determined by the precision of the arguments? So, for example, your program might contain:
r = s15adf(x,ifail)
If x were single precision then a single precision would be returned to r. If x were double or quadruple precision then the corresponding double or quadruple computations would be performed.
I can imagine a number of scenarios where this facility might be useful, but I invite your comments on how common these might be:
  1. Single precision might be useful as once again single precision calculation becomes faster than double precision. So the user might want to trade accuracy for speed.
  2. Secondly, single precision used to be the natural interface to many graphics packages. I am not sure whether this is still the case. Would it be useful to have the capability for this reason?
  3. Double precision, the precision now offered, is generally ‘about right’ for most computations, but sometimes extra precision is needed for the trickiest calculation. Under these circumstances quadruple precision might be necessary. How common is this requirement amongst our users?
  4. If a computation is made with two different precisions and the resulting answers are broadly similar then a certain level of confidence might be gained from these computations. If on the other hand the results are very different this may indicate that the original problem was ill-conditioned. The ability to do this test easily might be something our users want. Do you think this is so?
From the developers point of view modern Fortran makes it easy to provide these facilities:

Module dks1

      Interface dks
        Module Procedure dks_1, dks_2, dks_3
      End Interface dks

    Contains
      Subroutine dks_1(x)
        Real (Kind=1) :: x

        Print *, 'single precision'
        Return
      End Subroutine dks_1

      Subroutine dks_2(x)
        Real (Kind=2) :: x

        Print *, 'double precision'
        Return
      End Subroutine dks_2

      Subroutine dks_3(x)
        Real (Kind=3) :: x

        Print *, 'quadruple precision'
        Return
      End Subroutine dks_3

    End Module dks1

    Program main
      Use dks1
      Real (Kind=1) :: x1
      Real (Kind=2) :: x2
      Real (Kind=3) :: x3

      Call dks(x1)
      Call dks(x2)
      Call dks(x3)
      Stop 'ok'
    End Program main

(Here the Kind convention is the natural one for the NAG Compiler; others might use a ‘Byte convention’ i.e. the kind number matches the number of bytes of storage. Thus some compilers use ‘Real (Kind=4)’ to denote single precision and ‘Real (Kind = 8)’ to denote double precision.
My implementation colleagues will point out that instead of checking over 3000 results per implementation they would have over 9000 if we were to provide this capability. How appreciative would our users be if we were to do this? I am sure that I wouldn't manage to convince my colleagues to take on this extra work without evidence that our customers would like it. So if you think this is worthwhile then please let us know. 

 

3 Years Later

$
0
0
Do you remember those episodes of Friends that consisted entirely of flashbacks - presumably the cast were too busy with other duties that week so couldn't film, or the writers too busy to think up a real story ...

The NAG Blog has been running for just over 3 years now. In that time we have had 118 posts on a wide variety of numerical computing topics including HPC, .NET, customer engagement, GPUs, optimization, and much more.

The most popular posts (pageviews) have often been the most technical ones - including:

On the HPC theme, there have been posts discussing factors of 1000x in performance, describing supercomputing to friends, family and politicians, and the relationship between supercomputers and ice cubes.

I hope you have found our blog to be a nice mix of useful content, occasional fun, and topical thoughts. Do you have a favourite NAG Blog post - or more importantly - is there a topic you would like one of our numerical computing experts to write about? From talking to customers we know that you’d like to see more technical articles which we are always working on, but we also would like to see more contributors to the blog. So if you have something to say about numerical algorithms or HPC that relates to NAG and the work that we do, get in touch and maybe we can collaborate.

Our other social networks – the NAG Linkedin Group and @NAGTalk on Twitter have grown to be fairly engaged spaces. Our aim with all our groups is to create a friendly space where users and staff can readily exchange ideas, innovate and share relevant stories. The Linkedin Group is a vetted group which means that we keep it free from spam and recruiters. Do join if you want to connect with other NAG users, collaborators and developers.

Evaluation of non-linear expressions using Excel

$
0
0
NAG provides VB headers for all the routines in the library allowing them to be called easily from Microsoft Excel given a little knowledge of Visual Basic. Along with the headers, some Excel examples are distributed with the product. You may also find demo worksheets for a variety of routines on the NAG web site.

Combining the ease of use of Excel and the power of NAG routines is a great way of creating simple, flexible interfaces for solving difficult problems. Typically you write VBA code to handle the reading and writing of data to the Excel worksheet and the passing of parameters to the NAG routines. Once this VBA code has been written the problem parameters can be easily changed and tweaked from the worksheet without having to alter any code again.

Routines which take only simple parameters such as scalar values, vectors and matrices can have all their arguments input via the Excel worksheet however for routines that require non-linear expression such as those routines with callback functions it isn’t so straightforward. One method is to simply write the callback in VBA however you lose the interactivity gained from using Excel as every time you want to change the problem you would have to edit the VBA code. This isn’t a big deal but there is another way!

The VBA function evaluate() can be used for just this purpose and, along with a few tricks, allows  natural input of non-linear expressions via the worksheet. The demo for the NAG optimization routine e04jcf takes advantage of this method. As you can see the objective function is expressed naturally on the sheet along with the simple input parameter required for the routine e04jcf:


Those with a keen eye will have picked up on the fact that the objective function is described in terms of X1…X4 but the initial estimation is given in cells B11…B14. The trick is that once the initial estimate has been read from the worksheet and stored in a VBA array, it gets copied back to the sheet in cells X1…X4 before the NAG routine e04jcf is called. Of course you could describe the objective function in terms of which-ever cells you like. It should be noted that this is not the most efficient method due to the extra copying of data that is required so it may not be suitable if your problem is large.

This technique can be exploited with many other routines in the NAG Library such as root finding, quadrature, systems of non-linear equations and many more. Once a worksheet has been created, it can be used by anyone who is familiar with Excel even if they do not know VBA.

Tip: You can hide column X by right-clicking the column header and selecting hide or set the font colour to white if you don’t want the intermediate values displayed.

The demo for e04jcf can be downloaded from our Excel page.

Calling the NAG C Library from kdb+

$
0
0
Kx Systems was founded in 1993 to address a simple problem: the inability of traditional relational database technology to keep up with escalating volumes of data. This problem is even more acute today, where data records, especially in the financial industry, number in the trillions. To provide users to fast data, the founders of Kx have developed kdb+ and vector-processing programming language 'Q'.

Kdb+ has the ability to quickly process data and for more complex analysis it has access to other programming languages. To call the NAG C Library from kdb+, a wrapper must be placed around the NAG routine, taking care when passing data types. The wrapped functions must then be compiled into an external library. Below is an example for calling a NAG regression routine:

#include<nag.h>
#include<nagg02.h>
#include<string.h>
#include<nag_stdlib.h>
#define KXVER
3          // Using version 3, this must come before #include "k.h"
#include"k.h"              // kx header file containing K structures

K regression(K x, K y)          //The two arrays we will input from kdb+
{
       double *wt=0;
       NagError fail;            
       INIT_FAIL(fail);
       K kout;                         //Array that we will return to kdb+
       kout=ktn(KF,7);            //Allocate kout - it will have 7 elements. KF indicates
                                          // it is an array of floating point numbers
       F*out=kF(kout);            //Set pointer to kout - comes after kout is allocated

       F*(x_input)=kF(x);         //Need to set pointers to input arrays into NAG
       F*(y_input)=kF(y);       
    
       nag_simple_linear_regression( Nag_AboutMean, x->n, x_input, y_input, wt, &out[0], &out[1], &out[2], &out[3], &out[4], &out[5], &out[6], &fail );

       if(fail.code != NE_NOERROR)
              return r0(kout),krr(ss(fail.message));    //Free memory and return any
                                                                         //NAG fail to Q
       return kout;
}

In production level code, we should really check array bounds and data types. If you would like, you can add the following checks:

     if(x->n != y->n) return krr("data not same length");
     if(x->t != -KF || y->t != -KF) return krr("input data wrong type");

We must then compile the above code into a sharable library:
gcc -shared -fPIC nag_regression.c -I/[NAG_Directory]/include  [NAG_Directory]/lib/libnagc_nag.so -o nag_regression.so

Before we start the Q language, make sure the library can be located:
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:[path_to_nag_regression.so]

Finally, we load the library in Q and test the NAG example program:
q)nag_reg:`nag_regression 2:(`regression;2)
q)nag_reg[1.0 0.0 4.0 7.5 2.5 0.0 10.0 5.0;20.0 15.5 28.3 45.0 24.5 10.0 99.0 31.2]
7.598166   7.090489   6.685763   1.322359   .8273433   965.2454   6

Outputs from the regression are the regression coefficients, the standard errors, the R-squared, sum of squares, and degrees of freedom (which all agree with the NAG example).

Interested in other NAG routines in kdb+? Email support@nag.com and let us know! Many thanks to the support team at Kx Systems.

ISC'13 Preview - Supercomputing in Leipzig

$
0
0
ISC'13 - the summer's big international conference for the world of supercomputing - is only a few weeks away. As usual, I will be attending, along with several of my NAG colleagues.

This year, ISC will be in Leipzig - a new location. That means as well as the #HPC [do I need hashtags in a blog post???] fun of the conference itself, we will all have to learn the foibles of a new host city. Which bar will replace the cornerstone that was the Radisson Blu in Hamburg? How quickly can we learn the layout of the conference center itself?

More to the point, what will be the big #HPC [there I go again] news stories and themes to watch out for in Leipzig? I have already been interviewed by InsideHPC as an early ISC'13 preview, and have written a blog for the official ISC blog (comparing #HPC and #F1) - but here are some further thoughts.

Take the MIC

Well, as far as big #HPC [getting silly now] news goes, I'm glad I didn't try to write this blog last week! I think the stand out news story of ISC'13 is pretty obvious (barring another dramatic reveal) - the excellent expose by HPC Wire of China's MIC (sorry Xeon Phi) based 50+ petaflops supercomputer. Yes, at the ISC'13 issue of the Top500 list, China's Tianhe-2 supercomputer will smash aside the USA's Titan supercomputer at ORNL to take the crown as the world's fastest supercomputer - by a huge margin.

Indeed, in a painful turn of events, Titan may be dumped out of the Top500 leadership position before it has even passed it's acceptance tests! I'll reserve further comment on the Tianhe-2 story for a upcoming blog post at my own HPC blog (hpcnotes.com).

I believe Tianhe-2 will not be the only Xeon Phi news story at ISC'13 - there will probably be a few other Xeon Phi based supercomputers to be announced or updated, along with several product announcements etc (including some from NAG).


Remember GPUs?

In my preview for ISC last year, I suggested ISC'12 would be a battle between MIC and GPU and other potential processor options (e.g. ARM, DSPs, AMD Fusion, ...). I think that will be true again at ISC'13 - in terms of a battle for press/attendee mind-share, but with the difference that Xeon Phi has probably taken a solid lead for the first time.

The problem for GPUs (in terms of excitement) is that they are not new anymore - people have tried heard about them, have tried them, have learnt CUDA, have found performance successes or not, and there has been plenty of dissemination. ARM still has the benefit of being the cool new player in this space - and while still young in terms of HPC product/market maturity - has shown great promise through various demonstration projects. Likewise, the AMD Fusion technology looks interesting.

NAG, of course, is please to be working with all of the above - in terms of providing training courses, programming services, HPC consulting, our Library products, etc.

Big Data

There are bound to be plenty of #BigData [hashtag!] themes emerging at ISC'13 - it is still in buzzword status after all! There may even be a few genuine case studies and success stories about the crossover between #HPC and #BigData. But I venture to suggest most mentions of #BigData at ISC'13 will be gratuitous rather than core to the topic at hand. Call me a cynic?

Use of HPC by industry

I have predicted that the HPC community will see a strong focus on industrial HPC this year, especially engagement between centers of HPC expertise and industry users. [Note that I say “centers of HPC expertise” – it is critical that this does not mean only supercomputer centers – there is a lot of real expertise in HPC outside of the supercomputer centers – e.g., within the main HPC vendors, or specialist HPC expertise providers such as NAG, or in some cases within the industrial end users themselves.] I would expect there to be plenty of conversation around this topic at ISC'13.

Surely we can't forget exascale?

Really, a #supercomputing conference without abundant #exascale discussion? The key question for ISC'13 is not whether #exascale will be discussed - it definitely will be - but what will dominate the #exascale discussion? The technical details and challenges of achieving exaflops performance (either peak or real applications)? Or the politics and funding issues? For example, the perceived risk to exascale funding in the USA, or the ambitions of China, Europe, and others towards #exascale?

The HPC ecosystem

I am a keen advocate that HPC is seen as a complete ecosystem of hardware, software, people, processes, etc. and not merely the hardware that is often the default focus of HPC. A supercomputer on its own is just a device for converting money into waste heat (via some floating point units and an oversized electricity bill). I'm hoping all parts of this ecosystem get appropriate attention in the conversations at ISC'13. (Read more on my views on this theme in the InsideHPC interview.)


What will you be looking out for at ISC'13? Leave a comment or get in touch for a direct conversation with me or NAG.

The NAG Library for Java – An Illustrative Example Using Black And Scholes Pricing

$
0
0
Introduction

Hello everyone, my name is Peter Guo. I come from Taipei city and work in NAG’s Taipei Branch Office. In this, my first NAG Blog, I want to present some applications of the NAG Library. I am pleased to be part of NAG and hope to contribute to the NAG Blog on many future occasions.

When I was a graduate student, I focused on researching numerical computation, derivative pricing, and financial management. My professor, Professor Lin, is a famous specialist on financial engineering and has been working in this academic area for more than 10 years. I am now delighted that he is now using the NAG C Library. He thinks that the NAG Library is not only accurate and fast, but also enjoys a good reputation for its routines, especially in optimization.

The NAG Library

NAG is a numerical algorithms company, providing high quality numerical computing software. The NAG Library contains algorithms which are powerful, accurate, and flexible to use, from operating systems as diverse as Windows, Linux, and Mac OS.

The next section contains an illustrative example of how to use the NAG Library for Java in which we use the problem of pricing a European option asa concrete problem to be solved. The problem is particularly simple, for which an explicit solution is known, however in the real world the problems are more complicated and alternative methods have to be employed. For this reason we use a very basic Monte-Carlo method to illustrate the use of the NAG random number generators, whilst acknowledging that in practice a number of variance-reduction techniques would employed by the professional.

Monte Carlo simulation

Monte Carlo simulation uses repeated random sampling to compute numerical results. It is used heavily in financial computation. In this article, I use quasi-random numbers (the NAG routine G05YJF) and pseudo-random numbers (the NAG routine G05SKF and the explicit Black and Scholes formula (the NAG routine S30AAF) to provide benchmark. Computation time and accuracy are measured to compare the performance of the two different random number generators.

The basic theory

The background to option pricing is given in the chapter introduction for the NAG special function chapter, S.


Assume that stock prices at time t follow the geometric Brownian motion as shown in the following stochastic process:












Given a function f(St,t) = logSt that is twice continuously differentiable, then the solution is

In the risk-neutralized pricing probabilities, equation (2) follows the stochastic process

Equation (3) may be discretized using Euler’s method :



Simulated stock price paths

Example 1: Generating simulated stock paths

Example 2: Generating the 95% confidence interval for stock price paths



























These results of a simulation using just two paths are arranged in table 1. We estimate the stock price at every time period by generating the random sequence from a Normal distribution. These prices can be shown in the figure 1. In the figure 1 the most simulated stock prices will be located between 95% upper and lower bound, however there are still very small number (such as those in the second path) located outside the 95% upper and lower bound. Breaching the lower bound is particular important because the portfolio is often concerned with downside risk.

Using the NAG routines

Example 3: Simulating the stock price paths (the NAG routine G05YJF). StockPriceSimulationQuasi.java
Example 4: Using pseudo-random numbers to simulate European option (the NAG routine G05SKF).   BlackScholesSimulationPseudo.java
Example 5: Using  quasi-random numbers to simulate European option (the NAG routine G05YJF).    BlackScholesSimulationQuasi.java
Example 6: Using the explicit Black and Scholes pricing routine (the NAG routine S30AAF).       BlackScholesCloseForm.java

Pricing a call option

The price of European option may be obtained by simulating equation (6).









The results for Monte Carlo method and Black-Scholes closed form are reported in table 2. Quasi and pseudo Monte Carlo methods are calculated by the NAG routines G05YJF and G05SKF respectively, and Black-Scholes closed form is calculated by the NAG routine S30AAF. Due to the fact that quasi-random number has uniform and dispersed sequence throughout the domain of space, this property makes them well-suited for Monte Carlo methods. They yield more accurate estimates than pseudo-random numbers for a given sequence length (Read the full introduction post here).

As shown in table 2, quasi Monte Carlo method may use fewer paths to converge to a true value (calculated here by Black and Scholes Closed form). The differences between quasi and pseudo Monte Carlo methods results diminish as the number paths increases, but the quasi Monte Carlo method is faster and has better relative error and standard deviation.

References

1. Black, F. and M. S. Scholes, “The Pricing of Options and Corporate Liabilities,” Journal of Political Economy, 81, 1973, pp. 637-654.
2. Boyle, P. P. “Options: A Monte Carlo Approach,” Journal of Financial Economics, 4 1977, pp. 323-338.
3. Using the NAG Toolbox for MATLAB – Part 3. (online)
4. G05SKF NAG routine document. (online)
5. G05YJF NAG routine document. (online)
6. S30AAF NAG routine document (online)
7. Klebaner, F., "Introduction to Stochastic Calculus with Application," Imperial College Press, 2005.

A quick but dirty way to use the NAG C Library Mark 23 from Python

$
0
0

I get requests about using NAG from python from time to time, and NAG strives to make its algorithms as accessible as possible from a wide variety of languages and platforms. Python is no exception.

While we don’t yet provide a fully documented python API replete with examples and so on, there is a quick but dirty way to get NAG C Library python bindings covering every library algorithm, struct, and enum.

My starting point was the old PyNAG, which is excellent as far as it goes. It is also for the older Mark 8 library -- which none of you out there in userland should be using.

PyNAG also looks a bit labor-intensive to produce for the whole library, which would entirely disrupt my Maynard G. Krebs Cycle. There has to be better way, I thought.

And there is! At least on Linux. For this exercise I used our NAG C Library Mark 23 implementation for Linux and gcc, LP64 model, hereafter abbreviated as cll6a23dgl.

Here is the method that I used: 

Step 1 -- Get the Tools

For this exercise we'll need two tools. The first is gccxml, the extension to gcc that produces an xml description of the gcc compiler's internal representation of a compiled code. The second tool is the python extension ctypeslib. Both are available from the Ubuntu repositories (for example.)

 

Step 2 -- Collect the C Headers

Make a file "nagall.h" that includes all of the relevant headers:

echo"#include <nag.h>"> nagall.h

find/opt/NAG/cll6a23dgl/include -name "nag[a-z][0-9][0-9].h" | sort | sed -e "s/^.*\//#include </" | sed -e "s/$/>/" >> nagall.h

echo "#include <nags.h>">> nagall.h

 

Step 3 -- Produce the XML

h2xml -I. -I/opt/NAG/cll6a23dgl/include nagall.h -o nagall.h.xml

 

Step 4 -- Produce the Python

xml2py nagall.h.xml -o nagall.py -l/opt/NAG/cll6a23dgl/lib/libnagc_nag.so

 

Step 5 (Optional) -- edits for clear C library concordance / compatibility with PyNAG

Edit the top of nagall.py to read:

from ctypes import *

 

#human edit

E04_DEFAULT = None

NAGCOMM_NULL = None

NAGERR_DEFAULT = None

NE_NOERROR = 0

#end human edit

 

and edit the bottom of nagall.py to read:

           'Nag_HashTable', 'Nag_SparseNsym_CheckData',

#human edit

'E04_DEFAULT','NAGCOMM_NULL','NAGERR_DEFAULT',’NE_NOERROR’]

#end human edit

 

Then with very minor modifcations the PyNAG examples will run. For many the only change needed is to change the import, from “PyNAG” to “nagall”.

Another necessary change is that Numpy arrays must be passed slightly differently than with PyNAG, e.g.

g01alc(n, x.ctypes.data_as(POINTER(c_double)), …

  

instead of

g01alc(n, x.ctypes.data_as(c_void_p), …

Nag C Library enum types are translated nicely and may be used like one normally would, e.g.

f07aac(Nag_RowMajor,n,nrhs,…

and callback function typing is taken care of for us, e.g.

def py_obj_func(n,xc,objf,comm):

     objf[0] = exp(xc[0]) * (xc[0] * 4.0 * ( xc[0]+xc[1] ) + …

     return

c_obj_func = NAG_E04CCC_FUN(py_obj_func)

While the module nagall.py generated in this way is immediately usable on Linux, it is specific to the local installation. To get it working on anything else a small amount of work remains to be done. First, we have to remove the extraneous debris from gcc that finds its way into nagall.py. Second, we must omit some symbols exposed by the NAG C Library on Linux that are not exposed by all NAG C Library Mark 23 implementations.

Once that’s done, we get a module suitable for Linux, Windows, and OS X, available here. Updated PyNAG examples are here.

End Notes:

On Linux, if you’re instead using the NAG C Library ILP64 implementation CLL6A23DH, use this alternative module.

On Windows, I tested with the 64-bit Anaconda python distribution, which comes with all of the python bits needed to run the examples, including the graphical Rosenbrock example. An applicable NAG C Library implementation is CLW6I23DA. I found I had to add the folder containing the NAG DLLs to the system PATH environment variable.

On OS X, I tested with python 2.7.5 from python.org. A suitable NAG C Library implementation is CLMI623DG. In order to run the Rosenbrock example, I also needed to install numpy-1.7.1, matplotlib-1.2.1, and XQuartz , since I’m running Snow Leopard.

In all cases I’ve assumed the NAG C Library has been installed in the default location. If on your system it’s installed elsewhere, you’ll need to edit nagall.py accordingly.

Thanks to my colleague Brian Spector for testing on OS X Lion and for some suggested example changes. Thanks also to my colleague Mike Hooper, whose gentle questions suborned my inner Krebs.

nag4py v0.11

$
0
0
Brian and I went ahead and made python packages based on my last post.

The module name has been changed to "nag4py". Included are a few new graphical examples, as well as C library style long names.

For Windows(CLW6I23DA), Linux(CLL6A23DG), and OS X (CLMI623DG): nag4py-0.11.zip

For Linux(CLL6A23DH): nag4py-0.11-ilp64.zip

These are source distributions one would install in the usual way.

If you've installed the NAG C Library to a non-default location, you'll need to edit "__init__.py"

Alternatively for 64-bit python on Windows, one might use this zipped msi installer nag4py-0.11.win-amd64.zip.

How to worry about everything (and nothing)

$
0
0



Worry - To feel uneasy or concerned about something.
In my last post I wrote about managing your e-mail inbox (in the narrow sense) and, more broadly, managing your work and commitments. I ended that last post with this quote: "If you worry about everything, then you don't have to worry about anything." At least one reader suggested that I owed them an explanation and so here it is.

You'll recall that I challenged you (and me) to get our e-mail inboxes empty at least once per week in a systematic way, as follows:
Go through you inbox one message at a time and ask yourself "Is there an action required?"

If the answer is "No", then either delete as trash, file it elsewhere for later reference or file it in your "great things I'd like to do someday but don't know when" folder. Be ruthless.

 If the answer is "Yes" then answer the question "what's the very next action required?" and one of the following four things happens to it

·         If you can take the needed action in 2 minutes, do it now and either delete the e-mail or file it in another folder for later reference.

·         If somebody else needs to do what's needed, forward it and delegate

·         If you need to act on it on a specific day or day and time (e.g., a meeting) put it on your calendar

·         If you need to act on it but it's not time or day specific, put it on your task list. By the way, if it's really a project (i.e., has multiple steps) put it on your task list as a project and just note the very next step.
You're done! Everything that was in your inbox is now in the trash, filed for later reference, delegated to someone else, on your calendar or on your task list.

What you just did was, at least for the contents of your inbox, to “worry” about each item for a short amount of time in a systematic way. You made a conscious decision about each item and put it where it belongs – in the trash, in a folder for later retrieval, on your calendar or on your task list. In fact, you just did something even cleverer – you took things out of your head and put them into a trusted system. For many of us, this simple act frees the mind to focus on the “bird” (task) in hand rather than the dozen or more “in the bush” (still to be done).
The thoughtful skeptics among you might be forgiven if you are thinking “How does taking something out of my inbox and putting on a task list really free up my mind?” The answer lies in what you do with the contents of your task list and the other things that are on our minds but not in the e-mail inbox. I’ll talk about that next time when I define the “everything” in “worry about everything”.

Giving Credit Where Credit is Due Department: Many of the concepts and ideas mentioned here are things I’ve learned from years of trying to perfect my implementation of a methodology invented by David Allen called “Getting Things Done” or GTD. You can learn more at http://www.davidco.com/.

Jenkins comes to NAG

$
0
0
We build a lot of Fortran and C code as part of our Libraries and Compiler, and we have over 40 years experience in software engineering and in numerical analysis and algorithms. However, not being a large organization means that it can be hard to keep up with the vanguard of new SE methodologies and frameworks. There are some approaches though that seem a no brainer to have as part of the workflow in a modern software company. Convergent evolution can lead to their development and adoption without any prompting from the greater community.

For decades we've followed and benefited from many of the principles of continuous integration (for example, version control and build automation---but no longer with SCCS or shell scripts!) and we've been running a self-hosted and self-maintained manager for automatic builds since the early 2000s. But we began to find the system showing its age and straining at the seams. The build manager uses dumb ssh-ing to launch each build on a slave, so there is no communication channel between a build and the driver. The build indexer has become quite unmaintainable and is, shall we say, rustic: it resembles a web page from 1995.

NAG Library automatic build implementation reports

We don't have the resources to pump any significant improvements into these systems, so a couple of years ago we had a look around for replacements.

Buildbot is nice. We liked the visualizations it comes with out of the box, and some of us know Python so we felt confident we'd be able to set up the software. There's no way of configuring by a GUI though, which restricts for us the audience of potential maintainers.

Hudson seemed too Java focused to fit easily into our model.

Then we lost some personnel and so the investigation stalled. It remained 1995 for a few more years.

In 2011 we audited and codified many of our processes; in December 2011 we were awarded ISO 9001 certification. There was a renewed interest in enhancing the reporting that we use for measuring code health. At around this time we also made major changes to our release schedules. We needed the greatest amount of automation, the smoothest development process, the most informative reporting of health that we could manage.

In the time since we looked at Buildbot and Hudson we'd heard good things about Jenkins, a fork of Hudson (but no relation to Leeroy). I tried Jenkins out on some small projects and I saw good things. The GUI makes it a doddle to experiment with a set up. There are lots of nice, relatively-mature and well-maintained plugins for email notifications, tracking violations of user-defined coding standards, reporting on code coverage, Chuck Norris, ... So we set up a small group to prototype a new configuration for use by the whole company, and then, all going well, to port over the old system.

The NAG Jenkins is getting off the ground at the moment. Here are some things we've implemented and discovered (i.e., been burned by).


  • We want as many of the slave workspaces as possible to be on an NFS disk. In practice this means for all Unix-like slaves. With this configuration we can easily, at a low level, peek at the builds if necessary - using find, or whatever. Initially we went as far as using a single workspace for every such node: something like /users/jenkins/workspace/. This is bad! The Remote FS root for a node must be unique to that node. Otherwise every node that launches will unpack the Jenkins .jar files into the same directory. If there are different Java runtimes accessing these same files then sporadic End of File errors will occur (presumably because of the different runtime states becoming corrupted or confused).

  • As a fun exercise in getting to know more about how Jenkins plugins work and how to develop them, we wrote a warnings parser for the NAG Fortran Compiler and added it to the Jenkins warnings plugin.

  • Some of our projects are pretty monolithic. We were hoping to use the HTML Publisher plugin to publish our build reports for easy access from a job's page. As far as we could see from its source code, this plugin does recursive directory traversals and accumulates its findings in memory. On some of our older slaves, with the workspace on an NFS disk, this was just taking an intolerable amount of time. As a compromise a post-build Archive the artifacts is good enough for us instead.

  • There were resource problems when trying to report code coverage using gcov, gcovr and Coberatura as described by SEMIPOL. Our gcov-instrumented build generates 20,000 *.gcda files and 23,000 *.gcno files. This is too much data for gcovr to aggregate, which it tries to do by building an XML DOM tree in memory. In the end we just use lcov to scan the gcov files, and then we use the HTML Publisher to publish the report index.

  • Mac slaves need to run headlessly.

  • Controlling Windows slaves as a service requires you to modify the user rights settings (although, as Jenkins tells us on the configure page for such a node, we probably deserve grief if we use this launch method). The Windows user launching the slave needs to be assigned the right to Log on as a service: start secpol and add the user under Local Policies -> User Rights Assignments -> Log on as a service.

  • We have a version-controlled $JENKINS_HOME, which Jenkins itself backs up as a periodic job (although I understand that there's a plugin which does a similar thing). Inspired by Keeping your configuration and data in Subversion we use a parametrized job that runs
    #!/bin/tcshcd $JENKINS_HOME
    svn add -q --parents *.xml jobs/*/config.xml users/*/config.xml userContent/* >& /dev/null
    svn pd -q svn:mime-type *.xml jobs/*/config.xml users/*/config.xml userContent/*.xml
    echo"warnlog\n*.log\n*.tmp\n*.old\n*.bak\n*.jar\n*.json\n*.lck\n.owner"> myignores
    echo"identity.key\njenkins\njenkins.security*\nlog*\nplugins*\nsecret*\nupdates">> myignores
    svn ps -q svn:ignore -F myignores .
    rm myignores
    echo"builds\nlast*\nnext*\n*.txt\n*.log\nworkspace*"> myjobignores
    echo"cobertura\njavadoc\nhtmlreports\nncover\ndoclinks">> myjobignores
    svn ps -q svn:ignore -F myjobignores jobs/*
    rm myjobignores
    svn status | grep '!' | awk '{print $2;}' | xargs -r svn rm
    svn ci --non-interactive --username=jenkins -m "$1"if ($status != 0) thenexit 1
    endif
    svn st
    

  • The facility for multi-configuration projects seemed attractive for one of our jobs, which parametrizes especially easily over the source-code branch and target architecture. We need to have all builds in the matrix share the same workspace and to build in a shared checkout. By default each build in a matrix project runs in its own workspace, but as with a free-style project this is easy to configure: select Use custom workspace in the Advance Project Options of the matrix job, which will then uncover a Directory for sub-builds field. Also by default the master job performs the designated source-code step (e.g., Emulate clean checkout) but then so does each child job in the matrix! So all of the jobs in our matrix clash as they try to manipulate the checkout at the same time. There doesn't appear to be a way to customize this short of doing the correct work you want using explicit job commands. So for the time being we're going to maintain individual and extremely similar jobs for this matrix. With some strong command-line fu we can easily batch modify all the configurations for these jobs when necessary.

Currently we have a roster of 24 Library builds in different configurations, plus a further 20 or so jobs for additional tests of our development infrastructure and for bookkeeping. There are a few next steps we'd like to try out.


  • We want to see how far we can go towards configuring a full implementation of a Library, from a clean checkout all the way through to a final tarball to send for review and QA. The nature of some of the algorithms we are testing makes it difficult to enable entirely-automatic verification, but clearly but we should be able to use Jenkins to create a workable first approximation.

  • Our canonical sanity-checking Library build is made using the NAG Fortran Compiler. All the current Library builds in Jenkins run nightly, so we need to look at making the checking build more continuous, after each commit (modulo some quiet period).

  • Other than the main job page for our projects we don't have any clever notifications enabled for tracking or visualizing the health of the jobs. Hopefully we can sort out some nice graphics to show on a TV.

We're pretty excited about what we can do with Jenkins and what we can get from it. For automatic builds at NAG the future is looking bright; the future is looking blue (well, green).

A nag4py update

$
0
0
In a previous post “A quick and dirty way to use the NAG C Library Mark 23 from Python” my colleagues described a method for parsing the NAG C library headers to create Python interfaces using the ctypes modules. Since then we have written a parser in Python to do this, modularised the package into chapters and polished the examples a little. We now have packages for Windows, Mac and Linux available to download.

Python is a great language for getting results quickly. It's often referred to as a batteries included language due to it’s large standard library that includes modules for almost anything you may need. Add that to it’s super clean syntax and the Python Package Index and you can see why it’s so popular. Thanks to packages like Numpy, Scipy and Matplotlib, Python has grown significantly in the area of scientific computing and is gaining ground as the FOSS alternative to MATLAB ®.

Using higher level languages for prototyping such as Python, Matlab or VBA is an excellent way to initially model a problem. Our customers find it extremely beneficial to be able to call the same NAG Library when moving into the production phase and still get same results as the underlying code hasn't changed.


Python e04ccc example


The benefit of parsing the C header files and using gccxml to generate the Python interfaces is all 1700+ NAG Library routines can be called from Python automatically. The drawback with this approach is you don’t get the most pythonic interfaces. One solution is to wrap the ctypes interface with a pure Python interface for the functions you frequently use. Take the simple function g01eac for calculating the one or two tail probability for the standard normal distribution, it’s ctypes specification is as follows:

g01eac.restype = c_double
g01eac.argtypes =[Nag_TailProbability, c_double, POINTER(NagError)]

You will be familiar with the structures used such as NagError if you have used the NAG C Library before. If you want to use Python exceptions to catch errors from the Nag C Library you can find the associated codes and messages in nag_errlist.h. All the header files can be found in theinclude folder of your C Library distribution. Lets take a look at defining a more pythonic interface normal_prob to wrap nag_prob_normal:


from ctypes import c_double
from nag4py.util import NagError, Nag_LowerTail, Nag_UpperTail, INIT_FAIL
from nag4py.g01 import nag_prob_normal

def normal_prob(x, tail=Nag_LowerTail):
       """
       The lower tail probability for the standard Normal distribution
       >>> normal_prob(1.0)
       0.841344746069
       >>> normal_prob(1.0, tail=Nag_UpperTail)
       0.158655253931
       >>> normal_prob(1.0, tail=1)
       Exception: Argument tail has an illegal value: 1
      """
       x
= c_double(x)
       fail
= NagError()
       INIT_FAIL(fail)

       res
= nag_prob_normal(tail,x,fail)
       
if (fail.code ==70):
             
raiseException("Argument tail has an illegal value:", tail)

       
return res

def main():
       
print normal_prob(1.0)
       
print normal_prob(1.0, tail=Nag_UpperTail)
       
print normal_prob(1.0, tail=1)


if __name__ =="__main__": main()


NAG has interfaces for the majority of the languages that are often ranked in the 10 most popular  (TIOBE) and we can now add Python to that list. Would you like to see a fully pythonic NAG Library? We'd like know to know the demand for Python here at NAG so leave us a comment on this post or send in any thoughts to support@nag.co.uk.    

Viewing all 81 articles
Browse latest View live