FsCheck: A random testing framework

New Post: Unable to run C# NUnit examples

brianhv — Sat, 06 Apr 2013 01:42:48 GMT

The answer was buried in the NUnit documentation and the Resharper options screen. I had to copy FsCheck.NUnit.Addin.dll to the appropriate bin/addins directory. On my machine, that was at C:\Program Files (x86)\JetBrains\ReSharper\v7.1\Bin\addins. I had to create the addins directory there myself. It seems like a similar procedure should work for the standalone test runner but I couldn't get it to work.

I also changed a Resharper setting; not sure if it's necessary or not:
Resharper -> Options -> Tools -> Unit Testing -> NUnit -> Load NUnit Addins -> Always

New Post: Unable to run C# NUnit examples

brianhv — Sat, 06 Apr 2013 01:04:26 GMT

I downloaded the zipfile with the current source tree and successfully built it in VS 2012. However, the tests in the FsCheck.NUnit.CSharpExamples project fail in both the ReSharper test runner and in the NUnit GUI. In both cases, all three tests fail with the message "Test method has non-void return type, but no result is expected."

Is there something else I need to do to get this to work?

Source code checked in, #6fc5a7cf6a68

Kurt Schelfthout — Thu, 28 Mar 2013 15:59:24 GMT

Add more accurate Replay test. Also checks whether the reported seed is the same as the input seed.

Source code checked in, #28992e7b6c48

Kurt Schelfthout — Wed, 27 Mar 2013 17:32:26 GMT

Add Replay property to PropertyAttribute for easier replay/debug of failed test

Source code checked in, #f11a27ac51d8

Kurt Schelfthout — Mon, 25 Mar 2013 20:15:09 GMT

Add possibility to put ArbitraryAttribute on enclosing module in Xunit. This makes it easier to use the same generators for a whole bunch of tests grouped in a module.

Created Issue: PropertyAttribute: Search for Arb override on the module level too [16799]

kurt2001 — Thu, 21 Mar 2013 13:23:05 GMT

Avoids having to set Arbitrary on each PropertyAttribute

Created Issue: Add possibility to run with seed to PropertyAttribute [16798]

kurt2001 — Thu, 21 Mar 2013 13:22:07 GMT

So users can just copy-paste the printed seed into the attribute and run with that

Source code checked in, #2213c345eb0f

Kurt Schelfthout — Thu, 17 Jan 2013 03:12:44 GMT

Add Gen.sample

Created Issue: Support generation of arbitrary Decimal values [16797]

empeak — Wed, 02 Jan 2013 07:15:34 GMT

FSCheck.Arb.Default is missing the Arbitrary<Decimal> member.

New Post: Infinite loop on impossible gen constraints

jackfoxy — Tue, 18 Dec 2012 20:51:34 GMT

It may be useful to have a live example of a custom gen that can generate an infinite loop.

https://github.com/jackfoxy/RandomBitsSolution/blob/master/RandomBitsTest/FsCheckTest.fs

let rndByteSeqGen = 
    gen {
//        let! inclLower = Arb.generate  //http://fscheck.codeplex.com/workitem/16796
        let! inclLower = Gen.suchThat (fun x -> x < System.Byte.MaxValue) Arb.generate
        let! exlUpper = Gen.suchThat (fun x -> x > inclLower) Arb.generate
        let! count = Gen.suchThat (fun x -> x >= 1) Arb.generate
        let x = new RandomBits("7816001122100133")
        return (x.RndByteSeq (inclLower, exlUpper, count) |> List.ofSeq), (inclLower, exlUpper, count) 
    }

inclLower is the inclusive lower bound and exlUpper is the exclusive upper bound for generation of a random sequence of bytes. If inclLower generates System.Byte.MaxValue, then exlUpper can never be generated. This will happen on nearly half of the generation attempts.

Closed Issue: arbitrary infinite loop [16796]

kurt2001 — Mon, 17 Dec 2012 18:19:07 GMT

Commented Issue: arbitrary infinite loop [16796]

jackfoxy — Mon, 17 Dec 2012 17:42:39 GMT

I have a custom generator which appears to go into an infinite loop between the suchThatOption let binding in Gen.fs and stdRange / stdNext in Random.fs about half the time, and works successfully about half the time. Does this sound like a known issue?
Comments: I solved the situation. after posting this. I was generating one of the parameters in my custom generator without constraints, and it was possible to generate this parameter so that a subsequent suchThat constraint would be impossible to solve. My back of the envelope calculation predicts this situation should occur in nearly half the runs, which was consistent with my experience. I think somewhere in the documentation it may mention this situation, but now I have proven for sure if a custom generator can generate unsovleable combinations it will loop.

Created Issue: arbitrary infinite loop [16796]

jackfoxy — Mon, 17 Dec 2012 03:21:48 GMT

New Post: Typeclasses

kurt2001 — Fri, 07 Dec 2012 12:55:32 GMT

Hi,

Yes, in fact I have! There are two main uses of typeclasses in FsCheck. For one of them, I have considered replacing the current (runtime) implementation with an implementation at compile time. For the other one, I don't think it makes sense.

Use 1: map from type to a default generator for the type. I think using a static overloading mechanism here is not idiomatic, as it takes useful functionality away - the ability to change default generators for types at runtime. Two important use cases are - (1) specifying a number of generators scoped to a bunch of test methods, and (2) overriding a particular generator while keeping the default generator for the aggregate type. Say you have a record type (int, string, string, double), and you are fine with the default int and double generator but don't want empty strings. You can then just override the string generator, and keep using the default (reflection-based) generator for the record type. With static overloading you'd have to rewrite the record type generator, which is pretty tedious. Also you end up wrapping primitive types a lot to take care of special cases (NonEmptyString, NonZeroDouble, etc).

To sum up, I think making the whole thing dynamic has some benefits that outweigh the cost (i.e. no type safety so occasionally annoying errors) . I have tried to remedy the latter a bit by having lots of debug output when you override generators (it says what generators it adds, which ones it overrides, etc).

Use 2: map from result type of a test to a Testable (e.g. for unit the test fails it doesn't thrown an exception, for boolean if it returns true, etc). At one point I started refactoring that to the approach you mentioned, but as I remember I ran into ordering issues, it became unwieldy, and compilation became pretty slow pretty quickly. I am unsure if the latter would carry over to FsCheck users (I suspect not). In the end I gave up somewhere halfway through because since this use case is effectively completely contained inside FsCheck, it doesn't bring much to the table for users. It would however get rid of some initialization problems that can potentially still happen.

It's an interesting technique.

Hope that makes sense!

Best, Kurt

New Post: Typeclasses

mausch — Fri, 07 Dec 2012 02:56:23 GMT

Hi, I wonder if you have considered using a technique like https://code.google.com/p/fsharp-typeclasses/ to implement typeclasses in FsCheck.

I know this would be a pretty big change, perhaps not worth the effort. But it's definitely a closer approximation to Haskell's typeclasses (e.g. type-safe, no reflection).

So basically, just trying to pick your brain about the potential pros and cons of this hypothetical change :)

For example, it would seem to make it harder to infer generators via reflection for discriminated unions.

Cheers,
Mauricio

Commented Feature: Port and integrate smallcheck [11859]

kurt2001 — Fri, 09 Nov 2012 01:31:48 GMT

Smallcheck [1] is another testing framework for Haskell, that generates an exhaustive combination of values for properties. It may be possible to integrate this exhaustive generation with FsCheck. For example, it could be used to generate the "smallest" value for which a property fails, by first testing randomly and moving to exhaustive testing if a test fails. And it may be useful for some niche scenarios where the number of combinations is very small anyway.

[1]http://hackage.haskell.org/cgi-bin/hackage-scripts/package/smallcheck
Comments: Looks good. Have fun!

Commented Feature: Port and integrate smallcheck [11859]

alex_bogomaz — Thu, 08 Nov 2012 12:53:58 GMT

New Post: Generating data for use outside of FsCheck

jackfoxy — Sat, 20 Oct 2012 00:15:21 GMT

I'll take a look at that, thanks. I still think sample is very cool. I wanted to use FsCheck as a data generator on another project, but could not figure out how to do it. Now I can use FsCheck to generate data for any purpose.

New Post: Generating data for use outside of FsCheck

mausch — Sat, 20 Oct 2012 00:11:04 GMT

> Unfortunately this is not enough for me to properly debug.

Try overriding ToString() in the implementation of PhysicistQueue, or switch to a record type.

> I would like to step into one of the failing cases with the debugger, but how do I break into a failing case and not a successful case?

You could use System.Diagnostics.Debugger.Break() when the equality test returns false.

New Post: Generating data for use outside of FsCheck

jackfoxy — Fri, 19 Oct 2012 23:51:20 GMT

I've only started using FsCheck, so likely I am missing something. Here is my use case for sample. Some of my problem stemmed from not understanding FsCheck well enough. My original fsCheck code was failing:

fsCheck "PhysicistQueue" (Prop.forAll (Arb.fromGen physicistQueueIntGen)

...and showing me my generated tuple

FSharpx.DataStructures.Tests.IQueueTest.reverse . reverse = id:PhysicistQueue-Falsifiable, after 2 tests (0 shrinks) (StdGen (2139253415,295630954)):(seq [0; 0; 1; 2; ...], [0; 0; 1; 2; -1])

Unfortunately this is not enough for me to properly debug. I would like to step into one of the failing cases with the debugger, but how do I break into a failing case and not a successful case?

Anyway, sample lets me do this:

let y = sample 100 physicistQueueIntGen

//so I can set a breakpoint after a failure

let rec loop l =

match l with

| [] -> ()

| ((q : IQueue<int>), (l' : int list))::tl ->

let qList = q :?> PhysicistQueue<int> |> PhysicistQueue.rev |> PhysicistQueue.rev |> Seq.toList

if (qList = l')

then loop tl

else loop tl
loop y